Use Serverless SQL pool to easily and efficiently refresh Power BI datasets

Quoting Microsoft’s Synapse product page

Azure Synapse Analytics is a limitless analytics service that brings together data integration, enterprise data warehousing and big data analytics. It gives you the freedom to query data on your terms, using either serverless or dedicated resources at scale.

I quite agree with the bit…

Resiliency against random software or hardware glitches.

Spark is a general purpose analytics engine targeted mainly at big data scenarios. In most cases, Spark runs in a big data cluster where cluster manager would be responsible for resource allocation and orchestration. Classically this cluster manager is YARN on Hadoop but…

Running Spark Scala scripts in tightened environments

Disclaimer

The method explained in this post worked on Spark 2.2.0. Unfortunately, when I tried it on 2.3.2, it failed. So if your Spark version is beyond 2.2.0, you can read for fun but don’t expect a working solution. You can use SparkILoop similar…

Yousry Mohamed

Yousry is a senior engineer working for Telstra Purple. He is very passionate about all things data including Big Data, Machine Learning and AI.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store