For a number of years now, I’ve been dreaming of getting a pleasant portfolio to showcase my initiatives as a budding knowledge scientist. After virtually 1 yr of reflection, trials, failures and some successes, I created my first portfolio on GitHub Pages. Pleased with this private achievement, I wrote an article about it to share the fruit of my analysis with the neighborhood, accessible here.
This portfolio was created utilizing the mkdocs python package deal. Mkdocs is an excellent package deal for this sort of venture, however with a couple of shortcomings, the principle one being the overall lack of interactivity with the reader. The additional I obtained into creating my portfolio, the extra pissed off I grew to become by the dearth of interactivity. My constraints on the time (nonetheless true at the moment) have been to have all the pieces executed freed from cost and client-side, so the GitHub Pages resolution was completely suited to my wants.
The additional I obtained into my static portfolio, the extra the concept of getting a dynamic portfolio system popped into my head. My objective was clear: discover a resolution to create a reader-interactive portfolio hosted on GitHub Pages. In my analysis, I discovered virtually no articles coping with this topic, so I began on the lookout for software program, packages and code snippets to deal with this drawback.
The analysis query guiding this text is: how do I create a dynamic, full-client-side web site? My technical constraints are as follows: Use GitHub Pages.
About dashboarding package deal, I select to restrict myself to Panel from the holoviz suite, as a result of it’s an awesome package deal and I’d like to enhance my abilities with it.
For the needs of this text, I’ve looked for and located many roughly comparable options. This text is subsequently the primary in a collection of articles, the goal of which will likely be to current completely different options to the identical analysis query.
However what’s the purpose of getting dynamic Github pages? GitHub Pages is a really attention-grabbing resolution for group/venture presentation, 100% hosted by GitHub, freed from cost, with minimal configuration and no server upkeep. The power to incorporate dynamic content material is a robust means of speaking about your group or venture. For knowledge professionals, it’s a really helpful resolution for shortly producing a dynamic and attention-grabbing portfolio.
Holoviz is an thrilling and very wealthy set of pacakges. It’s an entire visualization and dashboarding resolution, highly effective on moderately sized knowledge and massive knowledge. This resolution helps all main enter knowledge manipulation packages (polars, pandas, dask, X-ray, …), and affords high-level syntax for producing interactive visualizations with a minimal of code. This package deal additionally means that you can customise the output and, specifically, to decide on your visualization back-end reminiscent of pandas (I’ve written an article about it should you’d like to search out out extra). To search out out extra about this nice suite of packages, I counsel this article.
For this job, my technical background imposes a couple of contingencies:
- I don’t but know the best way to code nicely sufficient in JavaScript to make full scripts and write items of code instantly in JavaScript,
- the dashboarding package deal will likely be Panel so as to enhance my abilities. If the necessity arises, I gained’t rule out repeating the train with different dasbhoarding packages (reminiscent of Sprint, strealint, NiceGUI, and so on.). Nevertheless, this isn’t my precedence.
For this text, my technical surroundings is as follows:
I exploit conda and VSCode for my scripts and surroundings administration. Don’t fear should you use different options, it gained’t have any influence on the remainder.
Throughout my analysis, I recognized 3 scripts of various complexity and visible enchantment from my researches, which is able to function good check requirements:
- A easy utility known as ‘easy app’:
import panel as pnpn.extension(design="materials")
slider = pn.widgets.IntSlider(title="Choose a worth", worth=10, begin=0, finish=100)
pn.Column(
"# Hey Panel + Quarto!",
pn.rx("You chose: {}").format(slider),
).servable()
- A extra complicated utility known as ‘huge app’:
import io
import panel as pn
import pandas as pd
import hvplot.pandaspn.extension(template='quick')
pn.state.template.title = 'hvPlot Explorer'
add = pn.widgets.FileInput(title='Add file', peak=50)
choose = pn.widgets.Choose(choices={
'Penguins': 'https://uncooked.githubusercontent.com/mwaskom/seaborn-data/grasp/penguins.csv',
'Diamonds': 'https://uncooked.githubusercontent.com/mwaskom/seaborn-data/grasp/diamonds.csv',
'Titanic': 'https://uncooked.githubusercontent.com/mwaskom/seaborn-data/grasp/titanic.csv',
'MPG': 'https://uncooked.githubusercontent.com/mwaskom/seaborn-data/grasp/mpg.csv'
})
def add_data(occasion):
b = io.BytesIO()
add.save(b)
b.search(0)
title = '.'.be part of(add.filename.cut up('.')[:-1])
choose.choices[name] = b
choose.param.set off('choices')
choose.worth = b
add.param.watch(add_data, 'filename')
def discover(csv):
df = pd.read_csv(csv)
explorer = hvplot.explorer(df)
def plot_code(**kwargs):
code = f'```pythonn{explorer.plot_code()}n```'
return pn.pane.Markdown(code, sizing_mode='stretch_width')
return pn.Column(
explorer,
'**Code**:',
pn.bind(plot_code, **explorer.param.objects())
)
widgets = pn.Column(
"Choose an current dataset or add certainly one of your individual CSV information and begin exploring your knowledge.",
pn.Row(
choose,
add,
)
).servable()
output = pn.panel(pn.bind(discover, choose)).servable()
pn.Column(widgets, output)
- A dashboard utilizing the ‘Materials’ Panel Template, which I name ‘materials dasbhoard’:
import hvplot.pandas
import numpy as np
import pandas as pd
import panel as pnPRIMARY_COLOR = "#0072B5"
SECONDARY_COLOR = "#B54300"
CSV_FILE = (
"https://uncooked.githubusercontent.com/holoviz/panel/principal/examples/belongings/occupancy.csv"
)
pn.extension(design="materials", sizing_mode="stretch_width")
@pn.cache
def get_data():
return pd.read_csv(CSV_FILE, parse_dates=["date"], index_col="date")
knowledge = get_data()
knowledge.tail()
def transform_data(variable, window, sigma):
"""Calculates the rolling common and identifies outliers"""
avg = knowledge[variable].rolling(window=window).imply()
residual = knowledge[variable] - avg
std = residual.rolling(window=window).std()
outliers = np.abs(residual) > std * sigma
return avg, avg[outliers]
def get_plot(variable="Temperature", window=30, sigma=10):
"""Plots the rolling common and the outliers"""
avg, spotlight = transform_data(variable, window, sigma)
return avg.hvplot(
peak=300, legend=False, colour=PRIMARY_COLOR
) * spotlight.hvplot.scatter(colour=SECONDARY_COLOR, padding=0.1, legend=False)
get_plot(variable='Temperature', window=20, sigma=10)
variable_widget = pn.widgets.Choose(title="variable", worth="Temperature", choices=record(knowledge.columns))
window_widget = pn.widgets.IntSlider(title="window", worth=30, begin=1, finish=60)
sigma_widget = pn.widgets.IntSlider(title="sigma", worth=10, begin=0, finish=20)
bound_plot = pn.bind(
get_plot, variable=variable_widget, window=window_widget, sigma=sigma_widget
)
widgets = pn.Column(variable_widget, window_widget, sigma_widget, sizing_mode="fastened", width=300)
pn.Column(widgets, bound_plot)
pn.template.MaterialTemplate(
website="Panel",
title="Getting Began App",
sidebar=[variable_widget, window_widget, sigma_widget],
principal=[bound_plot],
).servable();
To satisfy my objective of deploying each internet and GitHub Pages, I’ll check the deployment of every of the :
- on an area python server, generated utilizing `python -m http.server`,
- GitHub Pages.
Visualize the optimum operation that dasbhoard ought to have
Earlier than I begin testing, I must have a benchmark of how every utility ought to work in an ideal world. To do that, I exploit the native emulation operate of the :
panel serve simple_app.py --dev
Clarification:
- `panel serve simple_app.py` : visualiaze the dashboard
- ` — dev` : reload the dashboard every time the underlying information are modified (could require set up of a number of different packages, specifically to trace whether or not or not the underlying information have been modified)
Right here is the anticipated end result:
These visualizations will allow me to see if all the pieces is operating easily and inside affordable timescales throughout my deployment assessments.
First step : rework python script to HTML interactive script
The Panel package deal transforms a panel utility python script into an HTML utility in 1 line of code:
panel convert simple_app.py --to pyodide-worker --out docs
Clarification:
- `panel convert` : python script conversion panel package deal command
- `simple_app.py` : python script to transform
- ` — to pyodide-worker` : Panel can transcribe the Python utility into a number of kinds of assist that may be built-in into HTML productions. On this article, I deal with the output `pyodide-worker`
- ` — out docs` : output folder for the two information (HTML and JavaScript) generated.
Within the docs folder (‘ — to docs’ a part of the road of code), 2 information with the identical title because the python script and the extensions ‘html’ and ‘js’ ought to seem. These scripts will allow us to combine our utility into internet content material. This code conversion (from python to HTML to JavaScript) is made doable by WebAssembly. Pyodide is a port of CPython to WebAssembly/Emscripte (extra data right here: https://pyodide.org/en/stable/).
When you’re not conversant in WebAssembly, I invite you to devour Mozilla’s article on the topic. I’ll be doing an article on the historical past, scope and potential influence of WebAssembly, which I believe will likely be an actual recreation changer within the years to return.
First check: native internet server deployment
1. Emulate the native internet server with python: `python -m http.server`. This command will return an area URL to your browser to connect with (URL like: 127.0.0.1:8000).
2. Click on on the HTML script of our python utility
Data: when searching our information by way of the HTML server, to mechanically launch the specified utility after we open its folder, title the HTML and JavaScript information ‘index.html’ and ‘index.js’. Instance :
app/
|- index.html
|- index.js
When the app folder is opened within the native HTML server, index.html is mechanically launched.
Take a look at report for deployment on native html server:
- Easy app:✅
- Massive app:✅
- Materials Dashboard:✅
After testing every of the three functions listed above, this resolution works completely with all of them, with no lack of pace in loading and utilizing the functions.
Second check: GitHub Pages deployment
On this article, I’ll shortly go over the configuration a part of GitHub Pages on GitHub, as I described it intimately in my previous article.
- Warning from step 1: the ‘docs’ file internet hosting the HTML and JavaScript scripts should be named ‘docs’ and positioned on the root of the git repository. These are 2 stipulations for deploying functions on GitHub Pages. Neither the folder title nor its location may be modified.
- 2 prospects :
2.a. Rename the app information ‘index.html’ and ‘index.js’, and place them instantly in ‘docs’. This resolution will open the GitHub Pages of your repository instantly on the app,
2.b. Create an ‘index.html’ file instantly in ‘docs’, and add a path to your utility’s HTML file.
Right here is the content material of ‘index.html’ that I created throughout my deployment assessments:
1. <a href="https://petoulemonde.github.io/article_dynamic_webpages/simple_app_pyodide/simple_app.html">Easy app</a>
<br/>
2. <a href="https://petoulemonde.github.io/article_dynamic_webpages/big_app_pyodide/big_app.html">Massive app</a>
<br/>
3. <a href="https://petoulemonde.github.io/article_dynamic_webpages/material_dashboard_pyodide/material_dashboard.html">Materials dashboard</a>
Clarification:
- `https://petoulemonde.github.io/`: URL of my portfolio
- `article_dynamic_webpages/`: my working repo for this text
- `simple_app_pyodide/simple_app.html`: HTML folder/utility to open. Within the repo, the file is saved in docs/simple_app_pyodide/simple_app.html, however don’t point out ‘docs’ within the absolute path. Why this distinction between the file explorer and the hyperlink? GitHub deploys from the docs folder, ‘docs’ is its working root.
3. Push within the distant repo (in my earlier instance, the ‘article_dynamic_webpages’ repo).
4. Within the repo, allow the creation of a github venture web page. Within the configuration web page, right here’s the best way to configure the GitHub web page:
That is the place the ‘docs’ folder is important if we wish to deploy our utility, in any other case we gained’t have the ability to enter any deployment branches in ‘grasp’.
Take a look at report: Deployment on GitHub pages:
- Easy app: ✅
- Massive app: ✅
- Materials dashboard: ✅
Regarding resolution 2.b. : It is a significantly attention-grabbing resolution, because it permits us to have a static dwelling web page for our web site or portfolio, after which distribute it to particular dynamic venture pages. It opens the door to each static and dynamic GitHub Pages, utilizing mkdocs for the static side and its fairly design, and Panel for the interactive pages. I’ll most likely do my subsequent article on this mkdocs + Panel (pyodide-worker) deployment resolution, and I’ll be delighted to rely you amongst my readers as soon as once more.
Issues encountered
The dashboards examined to this point don’t distribute to different pages on the location/portfolio, so the one various recognized is to create a static dwelling web page, which redistributes to dashboards inside the website. Is it doable to have a website with a number of pages with out utilizing a static web page? The reply is sure, as a result of dasbhaords can themselves combine hyperlinks, together with hyperlinks to different dashboards on the identical website.
I’ve modified the Materials app code so as to add a hyperlink (including `pn.pane.HTML(…)` ):
pn.template.MaterialTemplate(
website="Panel",
title="Getting Began App",
sidebar=[
pn.pane.HTML('<a href="127.0.0.1:8000/docs/big_app_pyodide/big_app.html">Big app</a>'), # New line !
variable_widget,
window_widget,
sigma_widget],
principal=[bound_plot],
).servable();
This provides a hyperlink to the appliance’s aspect bar:
Whereas the proof right here isn’t fairly, it present {that a} dashboard can combine hyperlinks to different pages, so it’s doable to create a website with a number of pages utilizing Panel alone — good! In actual fact, I’m concentrating right here on the dasbhoarding a part of Panel, however Panel will also be used to create static pages, so with out even mastering mkdocs, you’ll be able to create websites with a number of pages combining static and dynamic parts.
Panel is a really attention-grabbing and highly effective package deal that allows you to create dynamic web sites simply and hosted on GitHub Pages, thanks specifically to the magic of WebAssembly. The package deal actually helps you to consider creating the dasbhoard, then in just some traces convert that dasbhoard into internet content material. Coupled with the convenience of use of GitHub Pages, Panel makes it doable to quickly deploy knowledge dashboards.
This resolution, whereas good, has a number of limitations that I’ve come throughout in the middle of my testing. The primary is that it’s not doable to combine user-editable and executable code. I’d like to have the ability to let customers discover the info in their very own means, by sharing the code I’ve written with them in order that they’ll modify it and discover the info in their very own means.
The second and remaining limitation is that customizing dashboards isn’t as simple as creating them. Hvplot affords, by way of the explorer instrument, a visible resolution for exploring knowledge and creating charts. However as soon as within the code, I discover customization somewhat troublesome. The packge is superior when it comes to energy and performance, and I’m most likely nonetheless missing a little bit of talent on it, so I believe it’s primarily because of my lack of observe on this package deal somewhat than the package deal itself.
When you’ve made it this far, thanks to your consideration! Your feedback on my earlier article have been very useful. Because of you, I’ve found Quarto and obtained some concepts on the best way to make my articles extra attention-grabbing for you, the reader. Please depart me a remark to inform me how I might enhance my article, each technically and visually, in order that I can write an much more attention-grabbing article for you subsequent time.
Good luck in your Pythonic journey!
Pierre-Etienne