Just lately I wrote an unconventional article about exposing analytics use circumstances in digital actuality. Although it was only a hackathon venture, it pushed me to consider what APIs (and through which kind) needs to be uncovered by headless BI platforms.

Once we discuss front-end growth, we often discuss Javascript/Typescript libraries. This was the case with the VR demo talked about above. However, particularly within the case of knowledge (analytics), Python language grew to become extraordinarily in style not solely on the again finish but additionally on the entrance finish. Some of the in style ecosystems these days is Streamlit.

An thought popped into my head: create an information software using a full set of APIs, which needs to be offered by headless BI platforms.

At present, some of the feature-rich information functions is the one permitting customers to construct stories (visualizations/charts/insights), so I made a decision to create such an software utilizing Streamlit and our Python SDK.

This text is backed by an open-sourced demo. It comprises not solely the Streamlit app but additionally a corresponding end-to-end information pipeline. It’s price mentioning that the demo lets you create a single pull request to ship every part constantly:

• Extract from information sources and cargo to the info warehouse (Meltano)
• Knowledge transformations (dbt fashions)
• Declarative definitions of analytics (GoodData)
• Knowledge functions (VR demo, Streamlit)

Why Headless BI?

We describe it right here.

Particularly, you possibly can join Streamlit on to information warehouses and even to information, however headless BI provides extra:

• Declare a semantic mannequin simply as soon as (logical information mannequin, metrics, stories, …)
• Join any shoppers (together with Streamlit), whereas counting on a single supply of fact
• Present low-enough latency to finish customers (scalability, caching)
• Forestall information warehouses from turning into efficiency bottle-necks or being too pricey

Answer

Let me spoil it right here and present you the total image first. This can be a screenshot of the ultimate software:

What are you able to see within the image? What am I going to speak about within the following chapters?

Use circumstances in self-service analytics!

Briefly:

• Semantic mode — offered within the left panel. Customers construct stories by choosing enterprise names. No SQL!
• Reviews: offered in the principle canvas. Numerous visualization sorts.
• Interactivity: filters, sorting
• Context consciousness – catalog is filtered based mostly on an already present report
• Multi-tenancy – change between a number of remoted workspaces
• Caching – each Streamlit and GoodData caching

If you wish to begin instantly with a hands-on expertise as a substitute of getting ready the entire ecosystem in your laptop computer, you possibly can attempt it right here.

In any other case, begin with the top-level README to organize information and analytics, then comply with it with the README for the Streamlit app to begin the app domestically.

Semantic mannequin

The demo repository comprises all of the details about how the semantic mannequin is generated.

We wish to expose the mannequin to finish customers within the Streamlit information software. Python SDK gives numerous capabilities for this goal. It’s doable to record every kind of entity – e.g. record attributes, info, metrics, and so forth. Moreover, it gives a perform to return the total catalog.

Furthermore, the SDK gives a perform to filter the mannequin by the already present report. What does it imply? If you put some entities right into a report, it may possibly restrict what different entities you possibly can mix them with. The mannequin consists of datasets related by relations. Not all datasets have to be related, and even when they’re, the route of the connection can influence the power to mix the entities.

Lastly, we wish to cache the catalog so we don’t name the backend with each web page refresh.

As an example, right here is the perform gathering the entire semantic mannequin (catalog):

Then, a Streamlit part like “multiselect” will be populated by catalog entities:

Helper capabilities are used right here to extract IDs and titles. Additionally, the Streamlit state is utilized right here to set the chosen values.

Report executions

Python SDK gives numerous choices on tips on how to execute stories. As a result of we’re constructing a Python software, it is sensible to make use of the Pandas extension, which may return Pandas information frames. They are often printed 1:1 in Streamlit or they are often straight handed as arguments to numerous visualization libraries offered by Streamlit, on this case, I exploit the Altair and Folium libraries.

We have to accumulate all the chosen catalog entities and fill them right into a report definition.

Each distinctive request is cached by Streamlit. It’s doable to clear the cache through the use of a devoted button within the left panel.

Metrics

Though GoodData gives an editor for creating metrics in a customized MAQL language (which is way simpler to make use of than SQL), the customers usually simply wish to create quite simple metrics like SUM(truth) or COUNT(attribute). The Streamlit software helps it, permitting customers to select a truth/attribute as a metric and for every to specify an analytics perform (SUM, COUNT, …).

Filters

The appliance gives an possibility to select an attribute as a filter. It’s doable to record all of the obtainable values for every attribute and show them within the Streamlit “multiselect” part.

Right here is how the attribute values will be collected from the server:

Although I carried out solely constructive attribute filters (attribute values equal to a number of values), GoodData, by way of Python SDK, gives many different varieties of filters out-of-the-box, e.g. destructive filters, metric worth filters, date filters, and so forth.

Sorting, paging

I made a decision to use sorting and paging within the Streamlit software, on the total end result set(information body). Nonetheless, GoodData helps sorting/paging out-of-the-box. Sooner or later, I wish to prolong the present answer accordingly.

Multi-tenancy

GoodData gives an choice to create remoted workspaces. It’s straightforward to help it within the Streamlit app — we simply record the obtainable workspaces, populate them to a devoted “selectbox” and let customers choose the workspace which they wanna discover.

Why Streamlit Rocks?

It’s very easy to onboard. Many constructing blocks are already carried out and simple to make use of, e.g. checkbox, multiselect, inputbox(textarea), and so forth.

Streamlit provides first-class help for state administration. It’s straightforward to persist much more advanced variables to state and entry them (after web page reload) utilizing dict or the property syntax.

It’s doable to cache even very advanced constructions. You simply merely use the @st.cache_data annotation and the results of the annotated perform is cached for every mixture of values of perform arguments.

Lastly, Streamlit gives cloud providing. Builders should register, after which they’ll create apps and bind them to GitHub repositories. Any merge to the repository redeploys the app with zero downtime. Cool! Furthermore, as soon as the app is displayed within the browser, it gives a developer console containing logs, settings, and so forth.

The place Streamlit Fails?

Though state administration is highly effective and simple to make use of, it’s typically difficult, particularly when you want to refresh elements based mostly on modifications in different elements, which is the case with catalog filtering. If you choose an attribute in “View by” you possibly can restrict the record of metrics. Essentially the most sturdy answer I discovered is to specify the “key” property of selectbox/multiselect elements. However, typically it didn’t work as anticipated and I spent hours discovering a workaround answer. That’s the reason the code is stuffed with “debug” calls, btw 😉

Concerning cache administration — the @st.cache_data annotation will be placed on class strategies, however it doesn’t work. I contributed to the corresponding Streamlit discussion board.

There’s a massive distinction between Javascript/Typescript apps and Streamlit apps – web page reloading. Each motion in Streamlit requires a full reload of the web page. Generally it’s helpful, however usually it’s not, because it doesn’t carry out. This can be a common limitation of the Streamlit structure, when every part is working on the Streamlit server, not within the person’s browser.

With rising latency between the Streamlit software and the GoodData, the appliance begins behaving weirdly in the course of the web page reload – e.g. the identical selectbox is displayed twice – as soon as lively and as soon as inactive.

Customized web page design is kind of arduous to attain. In my case, for example, I wished to create a high bar containing e.g. the workspace picker, however I didn’t discover a answer for it. There’s a corresponding difficulty opened for years.

Furthermore, a typical self-service analytics software gives a drag-and-drop expertise. Nonetheless, implementing this characteristic with commonplace Streamlit constructing blocks appears unattainable. Luckily, my colleague efficiently overcame this limitation by implementing a separate React software. This React software can simply be built-in with a local Streamlit app. I plan to jot down in regards to the integration in a follow-up article.

Lastly, I used to be unhappy that Gitlab is just not supported. What a pity! My pipeline advantages from Gitlab lots. To check the cloud deployment, I lastly pushed from the native to a Github “clone” repo, and it labored as anticipated. Personally, I might recognize it lots if it could be doable to set off the deployment from the pipeline, even earlier than the merge, to create a DEV surroundings, which can be utilized as part of the code evaluate. It might be excellent if the URL to such DEV deployment could possibly be put to the pull request as a remark 😉

So, Ought to You Use Streamlit?

Brief reply — undoubtedly sure.

Lengthy reply — undoubtedly sure, if you’re OK with the restrictions described within the earlier chapter. In any other case, Streamlit (and Python normally) gives a lot performance and so many libraries within the space of knowledge analytics/science. Personally, I’m most excited by the concept of blending the demo app I described right here with an embedded Jupyter pocket book(library exists), and offering a combined expertise for information analysts/scientists.

Try Headless BI for Your self

Able to expertise the facility of headless BI? Begin your 30-day free trial right now.