Another California City has moved its GIS Environment to the Cloud! The City of San Marcos has not only reduced the costs of maintaining on-site infrastructure but can also take advantage of cloud infrastructure's performance, security, and integration capabilities.
San Diego-based Esri business partner Quartic Solutions worked hand-in-hand with City staff to assess organizational needs and designed and built an Esri ArcGIS Enterprise Cloud environment running in Microsoft Azure. The City's primary goal, to relieve the strain on overburdened on-site resources, was achieved and exceeded, with a robust environment that will improve communication between people and multiple specialized municipal systems.
The City of San Marcos GIS staff is responsible for maintaining and providing high-quality, site-specific, authoritative spatial data to all departments within the City, including Planning, Development Services, City Clerk, Finance, Fire, Public Works, Parks and recreation, Storm Water, and Community Economic Development. Since 2006, the City has developed over a hundred different GIS data layers and maps that have supported both planning and operations needed for development services and other departmental business.
The City has been running its GIS hardware and software in an on-premise environment for over a decade. As more and more GIS functionality has moved to the cloud, it has become common for cities to move away from limited internal infrastructure to more flexible and reliable off-premise solutions. The City of San Marcos has done just that. The City’s miniature GIS web deployment has gone from a poorly performing, low-reliability mapping system to one that is stable and ready for the future.
The GIS database underwent a significant overhaul in the cloud migration project. The GIS database structures were streamlined, eliminating redundancies and enabling better data management. The database version is now 10.9.1.
The City also took advantage of this opportunity to retire its old GIS desktop client software, ArcMap. Migration is proving to be a success, and Esri’s ArcGIS Pro is proving to be a big jump in functionality and adaptability.ArcGIS Pro, the powerful single desktop GIS application, is a feature-packed software developed with enhancements and ideas from the ArcGIS Pro user community.
The City of San Marcos GIS staff is responsible for maintaining and providing high-quality, site-specific, authoritative spatial data to all departments within the City, including Development Services, City Clerk, Finance, Fire, Public Works, Parks and recreation, Storm Water, and Community Economic Development. Since 2006, the City has developed over a hundred different GIS data layers and maps that have supported both planning and operations needed for development services and other departmental business.
Quartic worked closely with the San Marcos CIO to develop a comprehensive design and implementation strategy for this specific Cloud implementation and upgrade project. The production rollout of the new system needed to go smoothly and without hiccups because there were existing system integrations supporting mission-critical applications. City crews use Lucity daily, The City's asset and work order management system. The City wanted a precise and automated rollout of the new system, so Quartic strategically automated the deployment process using a streamlined scripted solution.
The team considered two rollout script development paths: Esri's ArcGIS Enterprise Cloud Builder Tools or Microsoft's Azure Bicep. Both solutions would have been successful, but since the City's Information Technology staff have expertise with Azure DevOps Pipelines, the final decision was to use Azure Bicep.
With the City leading, the comprehensive rollout plan enabled Quartic to execute the scripts, resulting in a smooth deployment into production. The migration went seamlessly, with no data or configuration loss and no service failure to the users.
The architecture of the new solution required careful consideration to ensure that the foundation was set for the GIS application needs of the future. The team carefully planned all components, including the network, storage, database, applications, and security. This careful planning has set a strong foundation for growth in the future.
Focus was put on the new environment's stability, performance, and reliability while upgrading the software versions, migrating the third-party applications, and completing the ArcGIS Enterprise deployment.
Network: Azure Cloud is connected to San Marcos’ local network, and there is a solution for resolving website names between Azure and the on-site network. The Cloud networks consist of a central network for shared cloud services and another specifically for ArcGIS Enterprise.
Storage: Azure's newest storage system replaces San Marcos' aging on-premise file storage. All components leverage premium storage options and Azure and Esri tools to ensure data is safely backed up to multiple locations.
Database: Azure's managed database stores San Marcos GIS data, which can scale to fit their needs and is also cost-efficient. This component was configured to create data backups every 12 hours to protect the data.
Application and Web: A "multi-machine" ArcGIS Enterprise deployment model runs on the latest Azure editions of Windows and Linux. This model best fits San Marcos' desire for future growth. Automatic operating system updates are tailored to ensure that all GIS applications are consistently available and responsive.
Security: To enhance security for the end product, the team used Cloud-native security tools in Windows and Azure that continually monitor for potential threats following Microsoft's security recommendations.
The new infrastructure was built using Azure Bicep Templates and Azure Pipelines (DevOps). Esri PowerShell DSC was used for the automated installation and configuration of ArcGIS Enterprise. The team used Azure Monitor, Azure Recovery Services, and Azure Automation Update Manager for regular infrastructure maintenance, backups, and upkeep.
Having GIS in the cloud will reduce the reliability risk and costs of maintaining on-site infrastructure. By retiring two on-premises servers and instead leveraging the latest technology, the City will experience a faster and more stable environment, enabling City staff to do their jobs more efficiently.
The new deployment also ensures that there will be no recurring outages that waste valuable time and resources. The speed at which staff can access and edit GIS data is increased. The display speed of imagery has also significantly improved, making spatial data editing more straightforward and efficient. Staff can now take advantage of the performance, security, and integration capabilities provided by Cloud infrastructure.
The City of San Marcos and Quartic Solutions implementation team tackled technical and logistical challenges by working together. The result was an Azure-hosted complete deployment of ArcGIS Enterprise, replacing an on-premises ArcGIS Server and Enterprise Geodatabase. Also, the team implemented an editing and publishing workflow leveraging ArcGIS Pro.
The City of San Marcos and the Quartic team were pleased with the project outcome and look forward to continuing the collaboration in 2024. There is discussion regarding further improvements for the GIS program, for example, leveraging Esri's ArcGIS Monitor application for better performance monitoring and migrating data editing to hosted data store items.
"Apart from already leveraging one of their staff as “on-site” staff for the City of San Marcos, I am myself a former member of their staff and was familiar with their excellent team and leadership. I was 100% confident in their ability to deliver a fully functional, secure, and integrated Azure cloud migration product.“
- James Crandall, GIS Program Manager, City of San Marcos.
Team Quartic
Visit: The City of San Marcos,
Check out our partner Esri, to learn more about the industry.
The City of San Marcos wanted to have a way to share mapping data easily and directly with citizens. They developed an easy to use website that is now open to the public. Check it out here: City of San Marcos ArcGIS HUB
The data site hosts a full range of GIS data, most of which is available for download. To make it easier to find specific datasets, the data were grouped into categories. Data such as parcels, roads, jurisdiction boundaries, addresses, parks, trees and more can be downloaded with just one click.
The website, called the San Marcos HUB, has showcased the City's interactive web maps and web applications. The engine behind the site is Esri's ArcGIS Portal HUB application. The site was developed by the City with the help of a San Diego based woman owned small business Quartic Solutions . Quartic specializes in delivering GIS solutions to local government and utility organizations.
The City of San Marcos, CA Emergency Management Operations has implemented their Incident Status Dashboard using Esri's State and Local Government Tools.
The Incident Status Dashboard is a comprehensive web-based dashboard that offers information that may be used to make decisions and enables staff to give a more effective incident briefing.
Check out our partner Esri, and Quartic Solutions, to learn more about location technology and what you can do at your organization.
GIS can be used in many ways to improve the operations of cities and local governments. The City of El Cajon uses GIS as a tool to discover, maintain, and visualize its NFPA 704 Warning Placard data.
Whenever large amounts of hazardous materials are being stored and used at a business, warning placards are required. These placards act as an immediate warning system for emergency service personnel, helping them to identify the kinds of materials present and the dangers they pose.
The City of El Cajon uses Environmental Systems Research Institute's (ESRI) Enterprise ArcGIS products in daily operations. They have a small GIS team that expertly tackles all problems. The City's GIS team supports 911 response, Pavement & Sidewalk Management, Sewer and Storm Drain Maintenance, Traffic Data, Street Light, Signal, Signage, Capital Improvement Projects, Planning Data, Housing Information, Inspection data for Fire & Building Inspectors & PW Inspectors, Special Event Coordination, Park Assets, Street Furniture, and more. With all that data, they use GIS tools to help others with Data Sciences and Spatial analysis to clarify facts, visualize patterns, etc.
In August 2022, Quartic was brought in to streamline processes by automating repetitive tasks and modernizing the City’s enterprise GIS system. Quartic is available to provide GIS expertise when El Cajon needs GIS consulting services.
Heartland Fire & Rescue is a joint powers agreement (JPA) between El Cajon, La Mesa, and Lemon Grove, which creates a cohesive Fire Rescue/EMS service area. Quartic created a field solution that would provide fire inspectors a way to catalog NFPA 704 Placard information. The NFPA 704 regulations ensure public safety by clearly identifying businesses that house hazardous materials.
The ArcGIS Field Maps solution included a new feature service, a new web map, and a new mobile data entry form. The solution greatly simplified in-situ data collection for Heartland’s inspectors.
By taking advantage of the newer capabilities of Arcade in ArcGIS Field Maps, field inspectors are able to maintain placard data more accurately. Once the Quartic team published the new enterprise geodatabase feature class as an editable service and built a web map for field editing, new functionality was exposed to users.
Eliminating data entry errors in the field was a requirement of the project. The new Arcade Editor enables more complex and creative expressions and is designed to enhance data quality by using controls such as field auto-calculations and required fields.
The team at the City maintains a central database of valid addresses and wanted this same address dataset to be used when assigning a business address to a new placard location. In cases where a placard is located where there is not already a valid address, inspectors can manually add an address on the fly. This was implemented by writing an auto-calculate Arcade expression. Using the spatial function” closest”, the application finds the nearest existing address and populates the address field. If inspectors know the address calculated is incorrect, they are able to flag the record and type the correct address into another field. This flag is useful for both controlling the initial address entry and tracking potential address corrections needed in the primary address database.
The new mobile data entry application meets or exceeds the scope outlined by the client, and the new map viewer is a great visual tool with point symbology that matches the 704 placard standards to a tee.
Heartland Fire & Rescue inspectors are now equipped with the latest Esri field collection technology and can quickly and accurately survey businesses for compliance with hazardous chemical placard posting regulations.
Because the project was built using an enterprise geodatabase-based feature service, the companion viewer web application refreshes the map in real time. In addition, Heartland and El Cajon GIS staff are able to further QC the address data and amend the primary source address database when inspectors flag missing addresses. This is a great example of a project which benefits the City long-term by improving not only the hazard placard maintenance process but also improving the process for maintaining the City’s core address data.
"My favorite part of this project has been working with the Quartic team, who are always upbeat, professional, innovative, positive, and smart. The outcome of this project was just what we wanted, and it benefitted us on many levels."
- The GIS analyst at the City of El Cajon.
Read more about the collaboration of Quartic and the City of El Cajon:
Check out our partner, Esri and Quartic Solutions, to learn more about location technology and what you can do at your organization.
Providing highly-technical solutions to complex problems is something Quartic excels at. However, fast-response and general staff augmentation can be just as necessary, which was an immediate need for the City of El Cajon .
El Cajon’s small GIS team expertly tackles all problems thrown their way. Quartic was brought in to assist their efforts, reduce their individual workloads through automation, provide new capabilities, and upgrade/maintain their overall enterprise GIS system. Quartic provides Senior GIS Programmer/Analyst expertise available to support El Cajon’s GIS needs as they occur.
Several of the City’s ArcGIS products, on both the backend and frontend were upgraded. The City was supporting daily operations by using Environmental Systems Research Institute’s (ESRI) Enterprise ArcGIS System 10.7.1. Their work used to be done primarily in ArcGIS Desktop ArcMap and ArcCatalog, but now since ArcGIS Pro is available, those older user interfaces are slated for retirement in early 2026. Services published to ArcGIS Server were based on ArcMap runtime, which has been retired as of ArcGIS Server 11. The database was re-worked and is now in alignment with best practices better supporting versioned editing and logical groupings of data. Actively edited data that was in stand-alone file geodatabases are now stored in the centralized enterprise geodatabase.
ArcGIS Server was upgraded to 10.9.1 to access newer capabilities allowing the City time to re-publish all services with ArcGIS Pro runtime and prepare for the 11. x version migration.
A new SQL Server enterprise geodatabase was built and all existing data migrated into it from the old database. This database structure implements smarter versioning and avoids using feature datasets as storage folders. Almost all actively edited data now resides in the enterprise geodatabase, consolidating it into a single production editing environment.
ArcGIS Pro is now the go-to, and new attribute rules both maintain data quality and encourage program use.
With Quartic’s professional GIS support and Esri’s current software, the City has successfully resolved many legacy issues. The GIS team now has access to current software capabilities, has better organized and maintained data, and has renewed confidence that the GIS system is in good shape and more able to support City operations going forward.
The City of El Cajon has been working with Quartic to streamline, automate and upgrade many routine GIS data maintenance tasks. For the City’s GIS staff, the manual GIS maintenance was consuming too much of their valuable time. Since the work has been completed, the team has been able to be more efficient and put their energy towards other priority projects.
Municipalities often share GIS data with neighboring jurisdictions and regional government agencies. El Cajon is no exception. The City of El Cajon shares data with both regional GIS agencies: SanGIS and SanDAG . Keeping data current between government organizations can sometimes be a challenge. Quartic helped the City to improve these data sharing processes. The previously manually downloaded data from the SanGIS/SANDAG Regional Data Warehouse has been automated. Python scripts have replaced labor intensive processes for download, extract, and data enrichment. These scripts also perform data administrative tasks such as cleaning up disk space.
With all of these changes/upgrades, there are no longer questions about where the data is located. The team knows that all the SanGIS data is up to date at the beginning of each month and can rely on its currency and authoritativeness. The new database is more performant than it was before. The team learned that migrating from ArcMap to ArcGIS Pro has a bit of a learning curve, but that was quickly shaken off with the proper guidance.
With the initial upgrades complete, Quartic is now assisting El Cajon with several other projects that have been on the back burner. The variety of the GIS work is great and exactly what Quartic is established for; providing quality professional services for all kinds of GIS-related tasks. For example, the team has published the City’s historical imagery, created new enterprise feature classes with enforced data quality checks, published new services, and built web apps and experiences for both internal and public-facing audiences. Together the team developed a City asset dashboard, a FieldMaps application for fire inspectors, and a Survey123 application for bicycle registration, and helped update the City’s organizational theme for its ArcGIS Hub and web applications.
"The Quartic Solutions Team has really helped my GIS vision come to life here at the City of El Cajon. Quartic can take a project and run with it, they listen to what you really want, and then they create that dream that you have always wanted to do for your City.”
-The GIS analyst at the City of El Cajon
Check out our partner, Esri and Quartic Solutions, to learn more about location technology and what you can do at your organization.
Making Los Angeles a better place to live for everyone is not easy, but the Los Angeles Homeless Services Agency (LAHSA) is up to the task. Every year they organize a homeless count that gives us a better understanding of the current situation and allows the city and county to plan accordingly.
Once again, LAHSA successfully organized a team of volunteers to hit the streets and count the homeless in the Los Angeles region.
The annual survey was held over the weekend from January 24-26 and utilized the latest location-based software from the industry leader, Environmental Systems Research Institute (Esri).
Quartic Solutions, a local women-owned business, assisted with the geographic information system (GIS) technology.
This year the count went smoothly, with over 6,600 citizens and staff enthusiastically signing up to volunteer using an online sign-up system. LAHSA had mapped out a network of dispatch locations, ensuring that volunteers knew where to start their shifts and that all areas of the region were covered correctly.
Homelessness in California is a huge problem, particularly in large urban areas such as Los Angeles County. To better understand and address the challenge of homelessness, quantifying the number and extent of homeless people is critical. Once a year, LAHSA conducts the annual Point in Time (PIT) Homeless Count to provide needed data for allocating resources. The state and city programs use the information the count offers to implement more effective and targeted relief programs.
Finding and counting the homeless in a large urban area such as Los Angeles has its challenges. Taking advantage of lessons learned from last year, LAHSA set out to create an application for the volunteers that was easy to use and dependable. This year the data collection utilized the latest location-based technology collecting not only a count of homeless individuals but also which census tracts they were discovered in. Using a lightweight mobile application made data collection easy for volunteers, and organized for LAHSA.
"The goal of the count is to get an accurate picture of the homelessness in our communities. With this information, strategies can be developed to reduce and end homelessness."
- LAHSA
As part of Quartic’s commitment to combat societal ills and promote equity, Quartic Solutions took part in the count in two different ways.
Quartic was on the Esri team to develop a mobile application for the count.
The application developed by the team was based on Esri ArcGIS® QuickCapture. This simple application is the fastest way to capture field observations. It allows the user to count a homeless person or shelter just by pressing a button on their phone. The data captured included the location of the observation, the type of observation, and the ability of the user to view their results in an easy-to-understand dashboard format.
Several quality control features were also built into the application to make data collection trouble-free and more accurate. Having a visual of the boundaries of the area in which they were assigned kept volunteers constantly aware of the boundaries and reduced double counting. Finally, the application displayed a map to the collection teams, clearly showing a GPS track of where they had driven or walked. This valuable feature allowed the user to easily see if they had missed any road segments so they could go back and traverse any gaps.
Quartic also had two people, the company President, and Quartic's Director of Marketing, volunteering as counters on the night of January 25.
All volunteers' common goal is to commit to the community and make a difference. The ultimate goal is to make Los Angeles a better place to live for everyone.
Overall, the count was successful, and the QuickCapture mobile app collected the data seamlessly. The count was well-planned, and LAHSA is happy with the result.
According to LAHSA, in 2022, the number of homeless in the Homeless Count by the LA County Supervisorial District was 65,111. That increased from 2020 with the number of 1405 individuals. This year's results from the count are not published yet. According to LAHSA's just-released article, they will turn over the data collected to its data partners at the University of Southern California (USC). USC will perform the statistical analysis, and LAHSA expects to release it in late spring or early summer 2023.
The San Diego Geographic Information Source ( SanGIS ), a Joint Powers Authority (JPA), was able to improve processes used for maintaining the regional geographic information system (GIS) by implementing Esri’s current Parcel Fabric and ArcGIS Pro editing tools.
GIS Land Records in the San Diego Region are maintained by SanGIS joint powers agency. The SanGIS mission is to maintain and promote the use of a regional geographic data warehouse for the San Diego region and to assist in the development of shared geographic data and automated systems that use that data.
SanGIS allows the City and the County to combine resources to meet common objectives to reduce duplication of efforts, maximize resources, provide for an efficient method of sharing information and provide timely updated data to the public.
The combined County and City of San Diego geospatial land base is massive. The dataset consists of over 17 data layers, including lots, parcels, roads, addresses, and open space easements. GIS Database maintenance, including additions, deletions, and updates, are made by 4 - 5 full-time editors. Having a team of trained editors enables the region to have the most updated land base possible.
SanGIS was formed in 1997 and is responsible for the maintenance of
SanGIS decided it was time to update the GIS maintenance environment to a new solution that would enable more effective workflows and streamlined editing. Quartic Solutions, a woman-owned Esri business partner and premier GIS services firm, was chosen and tasked with migrating 20 years of legacy code, tools, and data integrity checks to a brand-new environment. Quartic and SanGIS have worked together for many years on various GIS projects in the San Diego region.
SanGIS was using a GIS editing environment that was developed back in early 2000. While the legacy system was stable, it was lacking current ArcGIS Pro functionality and there was a desire to improve consistency and communication. The old editing system consisted of stand-alone feature classes with their own individual topology rules. Data integrity was maintained by a mixture of ArcObjects Class extensions and custom ArcObjects ArcMap tools. All editing was done in ArcMap directly against a local Oracle geodatabase.
There are many departments within the stakeholders’ organizations, such as Sheriffs, Fire, and Assessors, who all need to be able to update parts of the SanGIS geodatabase. Because many of the Departments have separate networks, in the past shared editing was accomplished via two-way replication of feature classes to an ArcGIS Enterprise deployment in the cloud. The external organizations could then edit via their own check-out, check-in replica workflows. While effective, this model led to a lag between updates being shared as well as communication problems.
The impending retirement of ArcMap and the ArcObjects SDK, along with the release of ESRIs focused Land records solution Parcel Fabric provided the opportunity for SanGIS to migrate to a modern editing environment: ArcGIS Pro in combination with ArcGIS Enterprise service-based architecture.
The Parcel Fabric, along with the required web feature service, allowed for the smooth migration of this legacy code and workflows to a more modern service-based architecture.
The Esri Parcel Fabric is designed specifically to handle many of the data quality control issues that previously had been implemented through custom programs such as ArcObjects tools and class extensions. Issues such as maintaining Tax Parcel history and external organization updates are a core part of the Parcel Fabric product, which simplified the upgrade tasks.
Quartic was tasked with migrating the legacy code, tools, and data integrity checks to this brand-new environment. Most of the dozens of old custom tools had out-of-the-box replacements. A good example of this was the incrementor tool. SanGIS had created a tool to sequentially update the assessor parcel number on new polygons, but this was an out-of-the-box tool in Pro.
Old approach vs. New approach.
Once prototyping and testing were complete the data migration models that Quartic developed allowed for the final migration to occur over a single weekend with no downtime.
A total of 17 feature classes and 5 standalone tables had to be migrated to the new environment. Most of these were related to the Parcel Fabric. The migration process used a combination of geoprocessing models and Python scripts to automate the migration.
During testing, users practiced editing using the new interfaces making sure they could take advantage of the new functionality while still achieving predictable and accurate results. There were Approx. 5 months of testing and data model refinement.
In an effort to reduce custom programming, the decision was made to not migrate any of the legacy custom tools. All workflows would be designed using out of the box ArcGIS Pro tools. Editors would then identify if there were functions remaining that really needed to be added via customization for the purposes of timesaving and data integrity. There are a few custom tools that editors identified that will be deployed in future phases of the project. To date, 4 required development, with a further of at least 1 one approved for the future.
Quartic delivered a new modern editing environment that utilizes the latest Esri software and architectural practices to SanGIS. The JPA seamlessly migrated from a 20-year-old environment to the next iteration of Esri’s GIS software without hiccups. The project ensures that SanGIS will continue to meet its goal of providing a current land records system for the entire San Diego Region.
SanGIS editors are quickly coming up to speed in using ArcGIS Pro and are impressed with the new tools and features available to them. They are also regularly discovering new Pro tools and features that allow them to edit more efficiently and accurately than in the past.
The web-based services have also been streamlined and have simplified the collaborative editing environment. The County Assessor can easily create their own Tax Parcel updates directly in the fabric. This has led to a substantial reduction in the duplication of work.
Customer feedback
"Since SanGIS went live with the new ArcGIS Pro editing environment, there have been no platform or system issues, and Parcel Fabric has been running smoothly.”
Program Manager at SanGIS
For more information about SanGIS: tchee@sangis.org
Check out sangis.org , our partner Esri and Quartic Solutions , to learn more about location technology and what you can do at your organization.
The City of San Diego has recently expanded its green waste recycling program to include weekly organic waste recycling collection for all households currently eligible for collection services. This is good news as it will enable the City of San Diego to get closer to its goal of reducing the waste that is being put into landfills. The City’s goal is to reduce waste going into its landfills almost entirely by the year 2040.
The Environmental Services Department (ESD) wanted to take advantage of smart technologies to make the task of managing the new green bins and greenery collection efficient. The technology employed includes three automated business systems: 1) work management, 2) Geographic Information System (GIS), and 3) fleet management.
The new weekly organic waste recycling collection includes curbside pickup of food scraps, food-soiled paper and yard waste. Some residents have already started receiving their new green bins and all neighborhoods should have curbside greenery pickup over the next several months.
Cities can substantially increase the life of their landfills by recycling green waste. The City has estimated that over three cubic yards of landfill are saved for every ton of greens recycled. With numbers like this it is obvious why in 2022 the City of San Diego began a significant expansion of its green waste management program.
The Environmental Services Department (ESD) in San Diego ensures that residents have a clean and safe environment by pursuing waste management strategies that emphasize waste reduction and recycling, composting, and environmentally sound landfill management.
Learn more about environmental services, recycling, and waste management here: https://www.sandiego.gov/environmental-services/geninfo .
The Environmental Services Department already had a Geographical Information System (GIS) in place to manage waste collection routes for refuse, recycling, and green waste and wanted to leverage that investment for the expanded organics pickup program. The GIS software ESD uses for collection is based on Esri’s ArcGIS Platform . The Department also had work management software, Salesforce, for managing the weekly schedules for bin pickup. The third, and new system added to the mix was chosen to provide fleet management capabilities. ESD chose Routeware as the additional software which is installed on in-cab units, gathers data from smart sensors, and cameras mounted on the waste collection trucks.
To reduce staff manual labor and increase efficiency, the City of San Diego needed an integrated interface that would pull together data from these three specialized systems. To assist with the integration, Quartic Solutions , a San Diego-based woman-owned GIS services company, was brought in. Quartic’s job was to implement an automated process for keeping data current between these systems. As Salesforce, ArcGIS, and Routeware are all modern technologies, they all utilize web service technology for communication. Quartic was responsible for creating the web service components (ArcGIS feature and map services) that bring Salesforce and Routeware data together.
By using web services, the team was able to design and construct a smart system architecture that would meet all the requirements of the drivers, supervisors and office staff. The team redesigned the previous Extract Transform Load (ETL) job to handle the complex table structures needed to keep the location site data, the bin container data, the route data, and fleet data organized and in-sync. They also created a REST endpoint for Routeware so the truck drivers can update the routing Information themselves.
The batch data synchronization process begins with Salesforce updating records in multiple GIS tables via an ArcGIS feature service. GIS synthesizes information and performs geoprocessing at the container level, the smallest unit among multiple tables, and brings in additional information from a variety of reference layers. The intermediate product includes the truck routing information which is sent to Routeware also via a feature service. Once Routeware reads data from GIS, routes are displayed in RCC for the drivers to view and update. After drivers pick up all the bins and make route attribute changes, the updated route information gets back to the GIS database via the same feature service. GIS performs another batch of geoprocessing to update individual polygons that represent routes per each bin type and then converts the container-level data back to the site-level data and sends the updates to Salesforce. Salesforce pulls the GIS delta via a feature service, and the department accesses the dataset via either the map service or Enterprise database connections.
Integrating complex systems can be tricky. The new system had to manage much larger datasets than in the past. To keep data in sync between the three systems, ETL jobs are scheduled to run at night when system usage is lowest. The Salesforce, GIS, and Routeware batch jobs are also scheduled to run in a sequential order at specific times, nightly. If any one of the batch jobs takes too long to complete it will likely have an impact on the subsequent job run. In order to improve performance, the team employed a solution called multiprocessing. This technique allows the GIS software to process large amounts of data within limited server resources. By using the GIS multiprocessing technique, the ETL job run time was able to be significantly reduced.
Then, to improve performance even more, the GIS was configured to take advantage of database structures, called database views. These database views speed up the processing by having the modified (aka delta) records in Salesforce and Routeware indexed and in-memory, greatly shortening the data pulling time.
The Esri ArcGIS technology used in the project was ArcGIS Pro, ArcGIS Enterprise and ArcGIS online / Portal.
The new system allows ESD access to the functionality they need to effectively manage the organic waste recycling program. Now, drivers can take advantage of in-cab tablets loaded with a wealth of quality spatial data. In-office staff can also be more productive by taking advantage of an integrated system of Salesforce, GIS, and Routeware. The benefits are anticipated to increase the productivity of staff and increase reliability and response times to the public.
The project was planned and scheduled to roll out in three phases. The first two phases are now complete, and the final one is estimated to begin by the end of the year.
Visit: The City of San Diego Environmental Services , Routeware, and please check out our partner Esri, and Quartic Solutions , to learn more about Spatial Technologies and how you can utilize them at your organization.
What is a State and Local Government Specialty?
Quartic Solutions has been an Esri partner since 2004. Esri offers these specialty badges for partners that have the qualifications for each specialty.
We are pleased to be recognized by Esri as a State and Local Government Specialty Partner. Partners in the newly launched State and Local Government Specialty Program have State and Local Government (SLG) expertise and the ability to drive value and return on investment (ROI) quickly by leveraging ArcGIS Solutions, or deploying COTS solutions in alignment with Esri’s Sector strategies.
Quartic Solutions has been providing ArcGIS platform-based solutions to state and local governments for nearly 20 years. Quartic has enabled many city, county, regional, and state agencies throughout the US to improve their operations by taking advantage of spatial technology.
The City engaged an experienced Esri Business Partner, Quartic Solutions, to do the initial planning and deployment of the City of Cupertino's ArcGIS Enterprise on AWS. The city moved from a single, internal server hosting both ArcGIS Server and SQL Server to a multi-service AWS deployment.
The City was able to significantly improve the daily processes for their mobile workforce and streamline the integration of their GIS and EAM in the field.
While Cloud computing is becoming the mainstream, many organizations are moving their on-premise GIS infrastructure to the Cloud. The Cloud offers many advantages to Location-Based applications by increasing the scalability, flexibility and deployment speed of underlying hardware. At the City of Cupertino, CA the decision to move their Esri Enterprise GIS system to Amazon Web Services Cloud (AWS) has been a winning proposition, and has reduced internal infrastructure administration costs, enabled better integration between their business systems, and increased the number of staff utilizing the multitude of new Esri Cloud based GIS tools, templates and solutions.
The City engaged an experienced Esri Business Partner, Quartic Solutions, to do the initial planning and deployment of the City of Cupertino's ArcGIS Enterprise on AWS. The city moved from a single, internal server hosting both ArcGIS Server and SQL Server to a multi-service AWS deployment. One of the main forces behind the Cloud migration was the the City's use of the Cityworks Enterprise Asset Management system (EAM). The City’s legacy architecture required field crews to either checkout data replicas each day, or maintain a continuous VPN connection from ruggedized laptops. Both of these approaches created were difficult to maintain and time consuming for field crews and also required significant technical support from both the GIS and IT Departments.
By making the decision to externalize the Esri map services and Cityworks geodatabases, the City was able to significantly improve the daily processes for their mobile workforce and streamline the integration of their GIS and EAM in the field. In order to alleviate security concerns, the City’s IT Department decided to use an Amazon EC2 instance in order to completely separate the Esri ArcGIS services needed from the City's internal network. This architecture eliminated the need for a reverse proxy and in turn, increased the overall security of the City’s infrastructure.
The project has been so successful that it rapidly grew in scale and scope. With the mobile workforce editing both GIS and EAM data securely in real time, in the field, the City’s utilization of GIS services grew. An RDS instance was added to handle the increasing database load and the scale of the EC2 instance was increased several times to accommodate a growing number of map services and users as new departments have implemented Cityworks.
The City’s former infrastructure consisted of a single, physical server running ArcGIS Server and SQl Server Express. The SQL Server instance contained two databases, a geodatabase and the Cityworks database for work order management. This server was on the City’s network behind its firewall.
Field crews used one of two different products to do asset management.
With the professional services of Esri Business Partner Quartic Solutions, the City stepped into the Cloud infrastructure. The first step was to migrate from the primary existing on-premise server to a publicly accessible replicated AWS EC2 instance. As with the on-premise server, the EC2 instance contained both the Esri ArcGIS Enterprise components and a SQL Server database.
Two-way geodatabase replicas were used to keep the internal geodatabases and public geodatabases insync. SQL Server contained two databases, one for Cityworks and a new geodatabase containing a subset of the feature classes of the main on-premise geodatabase. The subset consisted of all the public works feature classes that would be used and edited by Cityworks and the public works field crews. This allowed the field crews to update assets in the field while simultaneously allowing the GIS department to perform the more rigorous GIS edits when required.
The next step was to move the internal REST-based map services needed by the crews to the Cloud. Esri map services that had previously been used by Cityworks from within the City's network were recreated on the new ArcGIS Server install on AWS. The transition was smooth and immediate results were very positive. Field crews found their workflows greatly simplified, GIS needed to spend less time assisting with the mobile units, and IT was happy with the security model, as none of the internal servers needed exposure to the outside.
The use of both GIS data and the Cityworks EAM software continued to grow as additional City Departments migrated to the new architecture. In addition, the GeoSpatial Cloud architecture enabled the City to build more and more beneficial web applications and easily make them available to the public. For example, the City’s 311 implementation was a tremendous success.
As demand grew, the addition of an AWS RDS instance was required. The RDS instance is entirely within the AWS VPC enabling ArcGIS Enterprise and Cityworks to continue to control access.
With the boom in interest and spread of additional GIS solutions spurred by the Cloud Architecture, the City has recently modified the maintenance process to push a full copy of the on-premise master geodatabase to RDS nightly. This allows up-to-date city reference data such as Landbase and planning data to also be made available quickly and easily to the public through various smart city initiatives the the city has implemented such as tree information service and ESRI open data portal.
The City has also heavily utilised AGOL for internal reporting and metrics through a series of public and private dashboards. These AGOL services are easily able to pull City data from the AWS cloud into authoritative, focused apps used across the City.
The use of both GIS data and the Cityworks EAM software continued to grow as additional City Departments migrated to the new architecture. In addition, the GeoSpatial Cloud architecture enabled the City to build more and more beneficial web applications and easily make them available to the public. For example, the City’s 311 implementation was a tremendous success.
The City of Cupertino's migration to a cloud platform continues to be a big success and is popular with the stakeholders. There are future plans to expand the approach to include more integration with the city’s network through OAuth and single sign on.
Even for smaller-sized Cities such as Cupertino, migrating to Cloud infrastructure can have a profound impact on staff’s ability to take advantage of GIS technology. For Cupertino utilizing Cloud infrastructure has not only streamlined internal operations, but has also enabled the City to truly provide Open Data and Government Transparency to its Citizens.
The County of San Diego is one of the most award-winning and innovative government agencies in the United States (County of San Diego, 2020). With this in mind, it is not surprising that their most recent GIS upgrade had a strong vision of providing better public access to GIS data via open data collaboration using ArcGIS Enterprise.
The effort to upgrade the County’s GIS system to ArcGIS Enterprise was led by the County’s GIS Manager, Ross Martin. From his perspective, “The pursuit of open data, transparency, and collaboration in the GIS world is a rather large undertaking technology wise. If you don’t get ahead of it the risk of falling behind is too great.”
The primary intent of the upgrade from ArcGIS 10.2.2 to ArcGIS Enterprise was the potential for better regional collaboration and the sharing of hosted GIS data via Portal-to-Portal collaboration groups. This included the desire to closely integrate with one of the County’s primary data providers, SanGIS.
The San Diego Geographic Information Source (SanGIS) is a Joint Powers Authority (JPA) of the City of San Diego and the County of San Diego responsible for maintaining a regional geographic information system (GIS) landbase and data warehouse (SanGIS, 2020). To enable a close linkage, the County wanted the ability to interface their on-premise GIS data with an enterprise environment hosted by SanGIS.
“We needed a solution that would allow us easy and secure access to our enterprise environment combined with the capability of sharing our data across agencies,"
Ross Martin
For this, there was no better solution than ArcGIS Enterprise to get the job done.
The new enterprise environment for GIS was implemented by the County of San Diego’s IT vendor, Perspecta in partnership with Quartic Solutions. To ensure County operations are available 24/7 the system was designed with high-availability, security, and ease of access as essential features.
“The best system is one that I can turn my back on,” explains Roberto Rivera, application architect for the project. To build a trustworthy system it was important that all of the network and hardware components had failover and redundancy place. This also required ensuring that these components could integrate properly with the ArcGIS Enterprise platform.
Quartic Solutions worked closely with the Perspecta team to ensure that the design leveraged all of the powerful features of the ArcGIS platform in regards to high-availability, data collaboration, and single-sign on (SSO) functionality. The next section details some of the design and technical considerations required to achieve these three goals.
Building a high-availability, multi-machine ArcGIS Enterprise required that the County’s third-party load balancer utilize application health checks for monitoring the uptime of Portal for ArcGIS and ArcGIS for Server. By default, ArcGIS Enterprise provides endpoints for detecting an outage of one or more ArcGIS components. Without this feature, a third-party load balancer may detect that a server is up without validating the health of ArcGIS. It was an essential feature that ensured that ArcGIS Enterprise could continue functioning in the event of a partial outage.
An equally important feature was the ability to provide a seamless single-sign on (SSO) experience for users that leveraged the County’s existing AD Federation Services implementation. Portal for ArcGIS offered a straight forward integration with ADFS with minimal configuration required. This functionality ensured that all County users could access the Portal from anywhere and without the need to type a password each time.
The ability for the County to connect to Portal organizations outside of their agency was crucial. For that reason, extra care and attention was put into ensuring that the Portal could function both internally and externally to their firewall. This also allowed for the ability to migrate their content from ArcGIS Online to Portal using Esri’s ArcGIS Online Assistant which allows users to search for and copy content from AGO to Portal. This tool was essential for streamlining their transition from AGO to Portal.
