Fast ingestion without compromising security
Data can be imported in three ways:
- Manually uploading local files (e.g., CSV, KML).
- Fetching files from various web services*. You can schedule this to run periodically.
- Using our standalone data integration service, and fetching
data from 20+ sources.
Whatever the method, illustreets enforces communication over secure channels without exception,
to mitigate the risk of eavesdropping. Our TLS setup achieves the A+ rating on SSL Labs.
Furthermore, data imports can be set to run over private connection, with AWS PrivateLink
or Azure Private Link.
When using our integration service, if expecting to connect to production databases,
we recommend installing it within a secure perimeter, and pushing the output to an encrypted
simple storage service with access logging.
If there are existing integration/transformation services in place,
such as Alteryx, we
suggest utilising an object storage service for unifying the output of these tools and
routing it to illustreets.
If required, maximum control over the data flow and complete privacy can be achieved by disabling manual uploads,
and limiting backend connections to the object storage service only.
* Out of the box, illustreets can fetch data from:
ArcGIS Server, GeoServer, Carto, Amazon S3, Azure Blob Storage, Dropbox,
Google Drive, Google Sheets. Also, any service offering direct download
over HTTP(S) is supported, e.g. Esri Open Data Hub, Natural Earth Data,
Data.gov, Socrata Data Platform, etc.
Protecting data in transit and at rest
In illustreets, the risk of unauthorized access is mitigated through a combination of
role-based access control (RBAC) and attribute-based access control (ABAC). This
capability allows you to fine-tune the team members’ access permissions:
who is allowed to access raw data, who builds visualizations, who can add/modify users,
who designs reports, and so on.
To protect against outside breaches and internal fraud, user actions are being logged
by the system in the form of an audit trail. Our strict HTTPS-only policy encrypts data during
transit at all times. On request, we can also enable database at rest encryption,
to further protect your data against intrusion and theft.
Ensuring data integrity and availability
Analysts are able to combine data, automate transformations and analysis, and link-up
data and visualizations all the way to the front end. Given this continuous data flow,
we enforce automated integrity checks, to minimize the potential for error and boost the
user experience.
Data is being continuously checked and validated against its
original schema automatically, throughout its entire lifecycle. This enforces document
similarity and prevents inconsistencies in front-end apps. Nobody wants a dashboard in the
boardroom full of NaNs or NULLs because some ETL process went wrong and mangled the source data…
When a dataset receives updates,
all its descendants — data views, maps, charts, and so on — are updated in succession,
all the way to the front end. This enables a degree of effortless automation rarely found
in other systems. Though, if you have data sources known to suffer from inconsistent quality,
you can create “data buffers” in the form of cached views, so the new data can be checked
manually before the updates are allowed to propagate further.
Standalone data integration service
We offer a data integration service, which can be installed on request. The options are: 1) allow full
integration with illustreets, whereby each analyst can create/modify sources via illustreets’ interface,
or 2) run it standalone and define a limited set of data sources within the service.
It can be installed on a separate network, on-premises, or in the cloud. You can choose to have the service
managed by your network team, or by illustreets. For increased security, you can consolidate the integration
workflow, and any other delivery pipelines you may be running internally, using an object storage service like
Amazon S3.
CONNECTORS
Out of the box, our data integration service supports these connectors. More can be added upon request.
Data warehousing
- BigQuery
- Redshift
- Snowflake
Database systems
- CockroachDB
- ClickHouse
- IBM Db2
- Microsoft SQL Server
- Mongo DB
- MySQL
- Oracle DB
- PostgreSQL
Web services
- Google Sheets
- Freshdesk
- Hubspot
- Quickbooks
- Salesforce
Static files - sources
- HTTPS
- SFTP
- SSH / SCP
- Amazon S3
- Azure Blob Storage
- Google Cloud Storage
- CSV
- JSON
- XLS(X)
- XLSB
- Feather
- Parquet
CSV files - geometry
- Polygon/line geometry column as WKT
- Point geometry as lon/lat or x/y columns
- Fully supporting the GeoCSV specification