Upgrade to the Modern Data Stack

 

Solution: The Modern Data Stack

There's already a tool perfectly suited to storing massive amounts of data, that can be queried easily, and is connected to everything. A database or data warehouse. You probably already have one running in your company that you can reuse so you don't need to buy another CRM, CDP, DMP, MAP, or any other acronym.

Building around a data warehouse has additional benefits such as:

• You own your data. It helps you comply with different regulations.
• Get value quicker. It is 10x easier to dump historical data in a DB than importing the data in yet-another-tool.
• Easier to sync with other tools. Databases integrate with everything, contrary to SaaS Tools that have limited APIs (and please don't get me started on APIs like Marketo).
• Reusability. Other teams in the company can use this trusted source of truth.

In addition to a data warehouse, you will need 4 other key components:

1. An event tracking tool. You can continue using Segment here. It does the job well and allows you to collect events across all of your websites & apps.
2. A data loader. I recommend Fivetran. It’s easy to set up in a couple of clicks and amazingly reliable.
3. A data modeling tool. DBT is the new power tool here. It allows you to transform and model your data.
4. An Integration Platform. I’m a 100% biased here, but I recommend using Census. We integrate well with DBT and enable you to sync your clean and unified data models back to all of your other tools.

As a bonus, you can replace Amplitude with a BI tool like Mode or Chart.io, which is cheap and as good as Looker.

Reference: https://www.getcensus.com/blog/graduating-to-the-modern-data-stack-for-startups

Modern data stack - 2021 and beyond

Modern Data Stack -  2021 and beyond

Product analytics and the modern tech stack

References:

https://mixpanel.com/blog/product-analytics-and-the-modern-tech-stack/

https://snowplowanalytics.com/blog/2021/05/12/modern-data-stack/

When to use Airbyte along with Airflow

 When to use Airbyte along with Airflow?

Airflow shines as a workflow orchestrator. Because Airflow is widely adopted, many data teams also use Airflow transfer and transformation operators to schedule and author their ETL pipelines. Several of those data teams have migrated their ETL pipelines to follow the ELT paradigm. We have seen some of the challenges of building full data replication and incremental loads DAGs with Airflow. More troublesome is that sources and destinations are tightly coupled in Airflow transfer operators. Because of this, it will be hard for Airflow to cover the long-tail of integrations for your business applications. 

One alternative is to keep using Airflow as a scheduler and integrate it with two other open-source projects that are better suited for ELT pipelines, Airbyte for the EL parts and dbt for the T part. Airbyte sources are decoupled from destinations so you can already sync data from 100+ sources (databases,  APIs, ...) to 10+ destinations (databases, data warehouses, data lakes, ...) and remove boilerplate code needed with Airflow. With dbt you can transform data with SQL in your data warehouse and avoid having to handle dependencies between tables in your Airflow DAGs.

References:

Airbyte https://github.com/airbytehq/airbyte

Airflow https://airbyte.io/blog/airflow-etl-pipelines

dbt https://github.com/dbt-labs/dbt-core

dbt implementation at Telegraph https://medium.com/the-telegraph-engineering/dbt-a-new-way-to-handle-data-transformation-at-the-telegraph-868ce3964eb4

Internal architecture and design of Snowflake!

Have you ever wondered how Snowflake has designed their elastic data warehouse, then here is an extremely nice read to know more details about it?

Reference: http://info.snowflake.net/rs/252-RFO-227/images/Snowflake_SIGMOD.pdf

What Tech Stacks need to be used?

Question: What Tech Stacks need to be used?

Answer: It depends!

Detailed Answer: 

It depends on various aspects of your functional and non-functional requirements. To know more, how this has been solved, then have a look at the following website stackshare to know what tech stacks are better for your specific requirements with a detailed decisions log from more than 1M developers community and with the information of tradeoffs and etc.

https://stackshare.io/feed

Open Source Miracles : NocoDB

NocoDB is an Open Source Airtable Alternative, and it turns any MySQL, PostgreSQL, SQL Server, SQLite & MariaDB into a smart spreadsheet.

It has the following rich features

• Smartsheet features such as search, views, filters, roles and permissions and etc
• Upload images to major storage cloud providers
• Workflow automation which includes alerts, notifications and etc
• Programmatic API features such as Swagger, GraphQA and etc

Reference: https://github.com/nocodb/nocodb 

Database migrations aren't tough; use the right tools

Database migrations are not tough and it's all about how you use the right tools to overcome your most common issues for your on-going database development changes.

Most famous open source tools are:

1. https://www.liquibase.org/
2. https://flywaydb.org/
3. Many more...

So, what to choose and when to use liquibase vs. flyway

• liquibase and flyway has more or less same features, except the flyway uses the SQL as a script whereas liquibase supports SQL, YAML, XML, JSON
• Easier setup and configurations then, flyway; liquibase also easy setup though
• Both flyway and liquibase have tooling and integrations support with your CI/CD tech stack
• Little more configurations with more features you need then, go for liquibase 
• liquibase has a lot of databases supported
• liquibase has database versions controlling
• liquibase has undoing the migrations, and doing the diff etc whereas, it's premium in flyway
• smaller teams and many projects then, go flyway
• bigger teams with many projects then go for liquibase as you can have rollback and diff features as all varieties of scripts can be supported

References:

https://medium.com/@ruxijitianu/database-version-control-liquibase-versus-flyway-9872d43ee5a4

https://github.com/flyway/flyway

https://github.com/liquibase/liquibase

Blockchain blocks and Merkle trees

A firm introduction on the blockchain can be found from here at https://blockgeeks.com/guides/what-is-blockchain-technology/ or https://en.wikipedia.org/wiki/Blockchain as a refresher... Now, quickly let me jump onto how these blocks data are getting verified with the right synchronization across all the nodes or blocks? How can we check the consistency, data verification, and data synchronization, etc?

The data structure behind it can be Merkle trees, Merkle tree also known as a hash tree is a data structure used for data verification and synchronization. It is a tree data structure where each non-leaf node is a hash of its child nodes. All the leaf nodes are at the same depth and are as far left as possible. It maintains data integrity and uses hash functions for this purpose.

The more uses of these Merke trees are below:

1. If you wanted to do data verifications across your distributed systems
2. DLTs (Distributed Ledger Technologies) such as Bitcon, Ethereum
3. Well known distributed databases like Apache Cassandra
4. Global supply chain
5. Health care industry etc

You may find more details of the Merkle trees along with sample code implementation can be found from the following sources:
https://www.geeksforgeeks.org/introduction-to-merkle-tree/
https://github.com/quux00/merkle-tree
https://medium.com/@vinayprabhu19/merkel-tree-in-java-b45093c8c6bd
https://www.codeproject.com/Articles/1176140/Understanding-Merkle-Trees-Why-use-them-who-uses-t

Do you need to refresh your 'data' of Master Data Management (MDM)

Do you need to refresh your 'data' of Master Data Management (MDM) experiences as a Data or Integration Architect, then just spend some time and read the following:

https://www.dataqualitypro.com/blog/beginners-guide-to-mdm-master-data-management

https://blog.stibosystems.com/the-complete-a-z-of-master-data-management

https://blogs.informatica.com/2020/02/10/master-data-governance-8-best-practices/

https://dwbi1.wordpress.com/2015/02/13/estimating-the-size-of-dimension-and-fact-tables/

https://profisee.com/master-data-management-what-why-how-who/

https://profisee.com/data-governance-what-why-how-who/

As an architect, a readily available template for doing the comparative tool's analysis
https://go.profisee.com/master-data-management-software-evaluation-template

Some more bonus tips from my experiences are:

To move faster with your Data Ops, you need to have well-defined data ops process with tools or platforms which suits your needs, it could be the data virtualization tools (for example, Delphix or Informatica, etc.)

Have buy-in from an executive sponsor within your organization to have a Data Governance initiative or forum who defines the policies, principles, master data hub technology & enabling technology, and data quality measurement

Also, choose a very good governance tool like Collibra (which can simplify your work for Data Governance related tasks at one place)

Focus on the following aspects during the Data Governance guild, such as

• MDM Strategy
• Governance groups
• Policies and rules
• Procedures and processes(life cycle of the data)
• Tools and systems
• Training and engagement
• Audits, and metrics 
• Measure and track the maturity, and then iterate

Bit more specifics on the functional Data aspects, such as

• Data modeling
• Data integration
• Data matching
• Data quality
• Data stewardship
• Hierarchy management
• Workflows
• Data archival and purging
• Data regulations and compliance with security requirements

As part of your MDM strategy, you need to look into all three pillars of master data management, such as

• Data origination
• Data management
• Data consumption

Comprehensive info along with comparative analysis and rankings on all Databases

Knowledge Base of Relational and NoSQL Database Management Systems

DB-Engines is an initiative to collect and present information on database management systems (DBMS). In addition to established relational DBMS, systems and concepts of the growing NoSQL area are emphasized.

https://db-engines.com/en/ranking

Java & Oracle Crawl Links...

If anyone wanted to get news and most happenings things on Java, J2EE and lot more, check the

http://www.javacrawl.com/

In the similar way, for Oracle tech stuff

http://www.oracrawl.com/

Above links are one stop shops for all the latest stuff around the world..!!!