So, yes this is selfhosting related. I am working on an n8n flow to pull in weather data so that I can have this data on a dashboard. I can’t find any dockerized weather forecasting apps. Most of them connect to a personal weather station, which might be an option in the future. For the time being, this is a little project I’m working on.
Partial JSON snippet:
spoiler
0
json
cod "200"
message 0
cnt 40
list
0
dt 1780693200
main
temp 29.4
feels_like 29.23
temp_min 29.4
temp_max 29.68
pressure 1019
sea_level 1019
grnd_level 984
humidity 42
temp_kf -0.28
weather
I would like to display something like this:
spoiler
Current temperature: 23.25 °C
Feels like: 24.09 °C
Low / High: 23.25 °C / 23.60 °C
Humidity: 94%
Atmospheric pressure: 1023 hPa (sea level: 1023 hPa, ground level: 988 hPa)
Temperature correction factor (temp_kf): -0.35
Weather: Light rain
Weather code: 500
Short condition: Rain
Icon: 10d
So, this is for you devs or coders out there. I can produce the JSON data. I’m just not sure how to parse it to something meaningful. I’m sure Python will have to be incorporated, but unsure of how to proceed.
Maybe someone could point me in a direction to tuts, articles, or your own experience. Sorry the JSON data doesn’t format correctly. Lemmy formatting doesn’t seem to allow that.
ETA: For anyone wandering across this, here is the solution: https://lemmy.world/post/47961985/24201369 thanks to @captcha_incorrect for his generosity and time.
You must log in or # to comment.
i use Nushell for this! works with JSON, YAML, TOML, markdown, Polars Dataframes, SQLite, and a bunch of others including builtin parsing tools for whatever formats and a plugin ecosystem. i use it at work and for personal projects as my main shell, and it’s super handy for exploring, unpacking, sorting, and visualizing all sorts of data. i use it to:
- find specific parts of YAML cloud configs
- visualize JSON logs, including a parser that restructures
journalctllogs. - _re_structure data from CLIs to work with them as structured: git logs, Unix coreutils, etc
- script my environment: common
kubectlqueries, specific web API helpers, building and running and testing applications, etc
it is a slight learning curve, and technically you could do all of that with
bashorzshandjqorjc, but i appreciate the modern take on your base shell terminal env.it’s replaced both Python and Bash for me.
Nushell
https://www.nushell.sh/? I’ll add it to this evening’s reading material. I was skimming the site and saw this: https://www.nushell.sh/cookbook/jq_v_nushell.html in relation to jq.
Thanks for the input.
exactly! i basically live in the terminal, and this is my go-to shell for all platforms
You can point the homarr iframe to a HTML file that consumes the n8n json output with JavaScript (fetch and transform data)
Even easier might be to point the iframe at the n8n endpoint but to add a HTML node as last node to your workflow. There you can use plain HTML and it directly supports the expression syntax to render the values from your previous json node.
isn’t json already a human readable format? ;) ok, jokes aside. to parse a file interactively, like i need to extract some information now, i use Julia, because I’m working with it anyway. that’s of course not very practical for automation scripts. most convenient would be a language that is already incorporated in your setup. otherwise i usually end up with a bash script with some
jqcalls.lots of ways to do it, where will it be displayed? if it’s in a console, jq would more than suffice.
if you want it to be processed heavily and added to an HTML document, probably something like a mix of jq and envsubst. you could forego all that and just use sed to inject the json into HTML then have JavaScript parse it out for you browser side.
if you want to store it in a database why not just store the whole json?
your requirements aren’t very clear on what you want this data for or how it will be used. without that there’s just too many variable solutions.
can you parse json? sure! what for/how will it be used? not sure.
hope this helps.
envsubst sed
Ahh more to explore. Thanks.
your requirements aren’t very clear on what you want this data for or how it will be used.
Apologies. As per usual, it’s all in my head and sometimes doesn’t make it to paper. I am using Homarr dashboard. Homarr has the ability to incorporate iframes. Ideally I would like to parse the JSON data into a prettier format perhaps adding some icons in the future once I get it all hammered out, and display it in an iframe on my dashboard.
Thank you for the leads and your time.
I don’t know much about homarr but it sounds like going with jq and envsubst might give you a faster solution that you can just iframe in your dashboard. though the learning curve can be steep if you’re unfamiliar with Linux command line.
I’m pretty sure you can literally pass in base64 encoded HTML to an iframe it will load in or pass it a publicly accessible file and it will work too.
I had messed around with building a locally hosted dashboard using similar methods so I’m pretty sure it will work.
Since you are using n8n, pull the data on a schedule and save it in a data table. Then create a webhook to serve the content as you want it. N8n can transform the JSON data, I can provide a sample tomorrow.
I can provide a sample tomorrow.
That would be swell, but I wouldn’t make it a priority. Just whenever you get a round to it. I have been given a ton of source material to read. Stayed up a little too late last night trying to assimilate it all. LOL
Sharing a workflow requires a paid plan but could should be able to paste this in the editor to get the same effect.
I made two different sample flows, on that call the api and responds with directly with the response data (transformed). The other workflow is to same the data in a data table, and call the weather api on a schedule. You could even use ntfy and send notifications on a schedule if it will rain within an hour.
Edit the nodes to suite your need (URL, content of response etc).
n8n sample workflow (copy and paste in n8n's editor)
{ "nodes": [ { "parameters": { "url": "https://opendata-download-metfcst.smhi.se/api/category/snow1g/version/1/geotype/point/lon/16.158/lat/58.5812/data.json", "options": {} }, "type": "n8n-nodes-base.httpRequest", "typeVersion": 4.4, "position": [ 656, 336 ], "id": "d47a17c9-9bc7-448c-841a-f3b2d787b2cb", "name": "Sample weather JSON data" }, { "parameters": { "jsCode": "let output = [];\n\nfor (const item of $input.first().json.timeSeries) {\n output.push({\n Time: item.time,\n \"Current temperature\": item.data.air_temperature + ' °C'\n // Add more as needed\n });\n}\n\nreturn output;\n\n" }, "type": "n8n-nodes-base.code", "typeVersion": 2, "position": [ 880, 240 ], "id": "cf4f11a6-0ecc-4e96-afcb-a7903113da19", "name": "Convert json to array" }, { "parameters": { "mode": "raw", "jsonOutput": "={\n \"Time\": {{ $json.timeSeries[0].time }},\n \"Current temperature\": {{ $json.timeSeries[0].data.air_temperature }},\n}\n", "options": {} }, "type": "n8n-nodes-base.set", "typeVersion": 3.4, "position": [ 880, 432 ], "id": "2f3765ba-6ee5-44b7-ac45-c5a407ed43e9", "name": "Manual mapping of one item", "disabled": true }, { "parameters": { "options": {} }, "type": "n8n-nodes-base.respondToWebhook", "typeVersion": 1.5, "position": [ 1104, 240 ], "id": "badd76a3-2920-4592-baa6-5202b4badba0", "name": "Respond to Webhook" }, { "parameters": { "operation": "upsert", "dataTableId": { "__rl": true, "mode": "list", "value": "" }, "columns": { "mappingMode": "defineBelow", "value": {}, "matchingColumns": [], "schema": [], "attemptToConvertTypes": false, "convertFieldsToString": false }, "options": {} }, "type": "n8n-nodes-base.dataTable", "typeVersion": 1.1, "position": [ 1104, 16 ], "id": "c76d9dfa-9ec1-421f-a96b-d1a65efb0dd7", "name": "insert (or update if exists) table with data" }, { "parameters": { "content": "Create the table first, then edit and select how to map the fields\n\n", "height": 224, "width": 320 }, "type": "n8n-nodes-base.stickyNote", "position": [ 992, -48 ], "typeVersion": 1, "id": "51347b11-fcb8-49db-bd17-405ec8719ec0", "name": "Sticky Note" }, { "parameters": { "jsCode": "let output = [];\n\nfor (const item of $input.first().json.timeSeries) {\n output.push({\n Time: item.time,\n \"Current temperature\": item.data.air_temperature + ' °C'\n // Add more as needed\n });\n}\n\nreturn output;\n\n" }, "type": "n8n-nodes-base.code", "typeVersion": 2, "position": [ 880, 16 ], "id": "3967c4c5-5443-40d2-b1fe-830aa9327118", "name": "Convert json to array1" }, { "parameters": { "url": "https://opendata-download-metfcst.smhi.se/api/category/snow1g/version/1/geotype/point/lon/16.158/lat/58.5812/data.json", "options": {} }, "type": "n8n-nodes-base.httpRequest", "typeVersion": 4.4, "position": [ 656, 16 ], "id": "f0043f80-c940-43cf-a5ef-2f7fe6dcc6a7", "name": "Sample weather JSON data1" }, { "parameters": { "path": "1572b231-edc5-4516-935c-f205b737c5d2", "responseMode": "responseNode", "options": {} }, "type": "n8n-nodes-base.webhook", "typeVersion": 2.1, "position": [ 432, 336 ], "id": "30953c4e-7627-4646-8b8d-ca46cc06c841", "name": "Webhook - Respond after API call", "webhookId": "1572b231-edc5-4516-935c-f205b737c5d2" }, { "parameters": { "rule": { "interval": [ { "field": "minutes" } ] } }, "type": "n8n-nodes-base.scheduleTrigger", "typeVersion": 1.3, "position": [ 448, 16 ], "id": "e242dea7-dc3c-4c0a-9630-786bf2a39b8a", "name": "Schedule to get weather data" }, { "parameters": { "path": "1572b231-edc5-4516-935c-f205b737c5d2", "responseMode": "responseNode", "options": {} }, "type": "n8n-nodes-base.webhook", "typeVersion": 2.1, "position": [ 448, -240 ], "id": "1c1ac8d1-5f64-4bab-a3ac-20f223b1411e", "name": "Webhook - respond with table data", "webhookId": "1572b231-edc5-4516-935c-f205b737c5d2" }, { "parameters": { "operation": "get", "dataTableId": { "__rl": true, "mode": "list", "value": "" } }, "type": "n8n-nodes-base.dataTable", "typeVersion": 1.1, "position": [ 656, -240 ], "id": "7433815b-229c-4ad8-88ce-2bcff688ece5", "name": "Get row(s)" }, { "parameters": { "options": {} }, "type": "n8n-nodes-base.set", "typeVersion": 3.4, "position": [ 864, -240 ], "id": "76cd6bfe-da95-41c0-a3ee-812956eab79f", "name": "Edit Fields" }, { "parameters": { "options": {} }, "type": "n8n-nodes-base.respondToWebhook", "typeVersion": 1.5, "position": [ 1072, -240 ], "id": "3c83ebc3-2c09-4d80-8278-e325237ae0e2", "name": "Respond to Webhook1" }, { "parameters": { "content": "No code behind these nodes, only a skeleton sample for the workflow could look like", "height": 208, "width": 944 }, "type": "n8n-nodes-base.stickyNote", "position": [ 384, -304 ], "typeVersion": 1, "id": "db35642a-04dd-427b-9a21-d4ad64dd5522", "name": "Sticky Note1" }, { "parameters": { "content": "## Via data tabel\n", "height": 592, "width": 1088, "color": 3 }, "type": "n8n-nodes-base.stickyNote", "position": [ 288, -384 ], "typeVersion": 1, "id": "877a2308-c70f-4b9e-8ef7-5dea94a812b8", "name": "Sticky Note2" }, { "parameters": { "content": "## Direct response from remote API", "height": 400, "width": 1088, "color": 5 }, "type": "n8n-nodes-base.stickyNote", "position": [ 288, 224 ], "typeVersion": 1, "id": "d06d7c02-a8eb-494b-8d10-142fb2c04037", "name": "Sticky Note3" } ], "connections": { "Sample weather JSON data": { "main": [ [ { "node": "Convert json to array", "type": "main", "index": 0 }, { "node": "Manual mapping of one item", "type": "main", "index": 0 } ] ] }, "Convert json to array": { "main": [ [ { "node": "Respond to Webhook", "type": "main", "index": 0 } ] ] }, "Convert json to array1": { "main": [ [ { "node": "insert (or update if exists) table with data", "type": "main", "index": 0 } ] ] }, "Sample weather JSON data1": { "main": [ [ { "node": "Convert json to array1", "type": "main", "index": 0 } ] ] }, "Webhook - Respond after API call": { "main": [ [ { "node": "Sample weather JSON data", "type": "main", "index": 0 } ] ] }, "Schedule to get weather data": { "main": [ [ { "node": "Sample weather JSON data1", "type": "main", "index": 0 } ] ] }, "Webhook - respond with table data": { "main": [ [ { "node": "Get row(s)", "type": "main", "index": 0 } ] ] }, "Get row(s)": { "main": [ [ { "node": "Edit Fields", "type": "main", "index": 0 } ] ] }, "Edit Fields": { "main": [ [ { "node": "Respond to Webhook1", "type": "main", "index": 0 } ] ] } }, "pinData": {}, "meta": { "instanceId": "139fc1c0283616d9fae3813879a161cf8e8e0992eea7a9e87dc980c7d4ed97ec" } }Oh Snap! You didn’t. I will definitely get this running in a bit when I finish all my chores and can go play. Wow man, thank you so much!! I had put this on my dashboard as a means of encouragement to finish what I started.
spoiler
ETA: That’s so purty. Haven’t fired it up, just imported. Got chores to do. Will report back.
spoiler
Just a follow up to first say thank you, and that the script is working for me. I’m doing a bit of tweaking but it certainly jump started the process. You’re a scholar and a gentleman.
Glad to hear that you got it working! The hard part for me was to figure out how to transform a JSON string into an array of n8n objects. I had already done that before so now I just copy that node whenever I need something like again.
I’m doing a bit of tweaking
I figured you would have. But that is half the fun. :-)
First of all, there might be more out-of-the-box tools out there for building dashboards that require little or no programming knowledge, but I’ve never used or looked for them. Microsoft’s Power BI is one that I’ve heard of in passing, but fuck micro$lop lol. It’s just the only example I can think of that might be more approachable. You could google something like, “open source power BI alternative” to find other options.
Here are a couple of tutorials about using Python to parse JSON:
- w3schools tutorial for using Python to parse JSON
- geeksforgeeks article explaining how to use Python to read JSON from a file
That will only cover how to interact with the JSON data, but not how to build a dashboard, ofc. Displaying the data in some fashion (e.g. a web page) is a whole other set of tasks.
If you’re new to programming in general, I recommend going through a collection of beginner tutorials on a programming language (Python is a good choice, but pretty much any will work for learning the basics). Any course like that will cover principles that apply to most languages and that would help you accomplish this task. For example, they’ll cover things like reading to and from files, parsing the content of files, and doing shit with the content.
Here are a few Python courses/tutorial collections you can check out. You don’t need to do all of them, but I wanted to give you some options so you’re more likely to find one that you vibe with:
- w3schools Python tutorial
- official Python tutorial
- Python Basics: Introduction to Python - this is one of Real Python’s “learning paths” which are structured like a full-on course
- I/O Operations and String Formatting - another course from Real Python. Goes more in depth on this particular topic and I would recommend this after learning the basics from one of the other resources I’ve mentioned.
EDIT: Btw, you are definitly not limited to using Python for this. Lot’s of other programming language will do. You just mentioned it, so I ran with that, haha.
If you’re new to programming in general,
I’m not new to programming, I’m just not very strong or proficient at all at it. You’ve given me a lot to read, and I thank you.
Btw, you are definitly not limited to using Python for this
I mentioned Python because that seems like a language a lot of people use for this sort of thing, but I am willing to entertain any other avenues…except perhaps Microsoft’s Power BI, which is discontinued.
Thank you.
Scoping first, code ideas at bottom. Kewl?
Is this a “learn it by coding it” project or is it a “I want this thing to exist, no one has done it, but my code skills aren’t quite there” project?
If the latter, would you consider iterating via llm (as you mention n8n, so I figure you’re in that space anyway) or is this a purely a learn by doing thing?
Come to think of it, there is actually a third option here. You could get the LLM to teach you how to code it by writing some pseudo code and asking it for pointers / starting steps. Claude web is pretty good for that sort of thing, I think. You can get it to tailor its lessons to what you need without the tedium of starting at “Hello World”.
You seem like the sort that could keep that interaction honest and not let it just do everything.
PS: I read about your setup - sounds brilliant. Go you good thing.
PPS: n8n has a Code node (JavaScript), and parsing that weather JSON into a formatted string is probably like 15 lines of code. Something like -
const data = $input.first().json;
return [{ json: { temperature: data.list[0].main.temp } }];
add a Code node after your HTTP request in n8n. Get one single value out first. If you see a temperature number in the output, you win.
Pulling Python in just to parse JSON is probably adding a tool you don’t need for this.
JavaScript may be ass but it’s literally there, so it’s omnipresent ass. :)
Once you’ve got the JSON parsed, turn it into one small HTML weather card.
const data = $input.first().json; const item = data.list[0];
const html =
<div> <h3>Weather</h3> <p>Current temperature: ${item.main.temp} °C</p> <p>Feels like: ${item.main.feels_like} °C</p> <p>Humidity: ${item.main.humidity}%</p> <p>Condition: ${item.weather[0].description}</p> </div>;return [{ json: { html } }];
(Sorry about the formatting ; Lemmy formatting is weird AF)
Anyway, as one journeyman to another, that’s where I’d start poking. ICBW.
Is this a “learn it by coding it” project or is it a “I want this thing to exist, no one has done it, but my code skills aren’t quite there” project?
This is a practical use learning project. I’m sure there have been many iterations of what I’m trying to learn. I’m a prolific reader and consume a lot of data, but I learn the best by hands on. Read, do, screw it up, rince/repeat ad nauseam until I get it right, and then I write that shit down because my brain is shit at remembering.
Perfect. Then my third suggestion stands - get it make you a bespoke training module.
Create a project container: upload the source documents already cited by others / ones you find useful. Just 4 or 5.
Add the rule “You are helping me learn ____. You must not provide full refactors unless specifically asked for. Use Socratic method where appropriate”.
Pseudo code what you want to do (hell, draw a flow chart) and ask “based on what I want to do, your background knowledge and the contents of the container, what’s the first thing I need to learn? How should I approach this?”
Then ask “why?”
Hell, get it to make you coding exercises.
After each session, get it to make you a handover note in markdown format. Download it and then add that to your container. (Later on, you can get really fancy and start making your own llm-wiki)
People shit on LLMs as a knee jerk thing… but coding IS a language…and if you understand the logic, the rest is syntax. Also, it’s not as if you’re making some million dollar mission critical thing- you’re teaching yourself through successive approximation.
If you don’t feel comfortable using online models, there are even local alternatives.
I still think JavaScript is the faster way for this, but who knows.
People shit on LLMs
Well yes, and both sides of the coin have valid points. I do use AI occasionally, and at the very least it gets me going in the right direction, tho not always 100% accurate.
