architecture – Running ad hoc queries on JSON log files

I have a situation where let’s say I have a folder called logs which has N folders.
Each folder contains events for a specific event type and each folder has N .log files where each file has multiple lines of JSON.

Example:

event1.1.log

{"id":1, "name": "ABCD"}
{"id":2, "name": "EFGH"}
{"id":5, "name": "IJKL"}
{"id":7, "name": "MNOP"}

event1.2.log

{"id":3, "name": "ABCD"}
{"id":4, "name": "EFGH"}
{"id":6, "name": "IFKL"}
{"id":8, "name": "ABED"}

Now, each event can have its own structure, but it’s guaranteed that each log in the same event will always have the same structure.

Now, I need a way to run ad hoc queries on these: get a list of students, get top ten students, etc.

I thought of loading them onto a temporary table and then run queries on it, but I was wondering if there was any other way to do this.

I could write an application that could parse the files in memory, but the amount of data could be huge to do computation in memory. And every time I want to run a different query on the same dataset within the next few days, it would have to parse all files into memory again.

Any approaches on this?

Running Program Again With Loop in Java

Running Program Again With Loop in Java – Code Review Stack Exchange

linux – How can I send a shell script to execute in a running Docker container?

I have about 24 Docker containers, each running a Java jar. One of my routine tasks is to connect to the host and run docker stats with a few formatting options to check on RAM usage, because every now and then some will start to creep up. When a container is using more than a predetermined percentage of its host’s memory, I then connect with docker exec -it <container id> /bin/sh and run the following:

  • jcmd to list PIDs
  • Identify the PID of the .jar process
  • jcmd <PID> GC.run to run garbage collection and get the memory usage back down

Naturally this can be a tedious and overly-manual process so I’m trying to streamline things a bit.

So far I’ve written this command which will do all of the bulleted steps at once:

jcmd $(jcmd | grep jarname.jar | awk '{ print $1 }') GC.run

Now I want to be able to send that from the host to the target containers, either from a file or just as the literal text. I’ve tried a number of different syntax suggestions from my search results and nothing seems to work. The closest I can seem to get is this:

sudo docker exec c4f7 jcmd '$(jcmd | grep jarname*.jar | awk ''{ print $1}'') GC.run'
Error parsing arguments: No command specified

Usage: jcmd <pid | main class> <command ...|PerfCounter.print|-f file>
   or: jcmd -l
   or: jcmd -h

  command must be a valid jcmd command for the selected jvm.
  Use the command "help" to see which commands are available.
  If the pid is 0, commands will be sent to all Java processes.
  The main class argument will be used to match (either partially
  or fully) the class used to start Java.
  If no options are given, lists Java processes (same as -p).

  PerfCounter.print display the counters exposed by this process
  -f  read and execute commands from the file
  -l  list JVM processes on the local machine
  -h  this help

where it seems to be calling JCMD but without anything else.

These containers are all running and need to stay running, so some suggestions I’ve seen in reference to starting a container with the command as an argument would not work.

For a one-tape Turing machine, Let BB(n) denote the largest T for which some n-state Turing Machine halts after running for T steps

I saw this question on Reddit, I forget the subreddit but it goes something like:
For a one tape turing machine with the input alphabet $Sigma$ and the tape alphabet $Gamma$. Let BB(n) denote the largest T for which some n-state Turing machine, started on a blank tape, halts after running for T steps.

  1. Why is BB defined? i.e why does the maximum exist
  2. Is BB complete?

signature – Sign rawtransaction with private key without running a btc node


Your privacy


By clicking “Accept all cookies”, you agree Stack Exchange can store cookies on your device and disclose information in accordance with our Cookie Policy.




running CLion on windows on different user

I updated CLion under a different user on Windows 10 and now I can’t start it because it gets an error:
enter image description here

It doesn’t even start when I “Run as administrator”, which is strange, because I am the administrator (I checked).

I’ve given myself full access to this executable via Properties -> Security -> Advanced -> Permissions but it still doesn’t work:

enter image description here

Then I’ve given myself full access to clion64.exe.vmoptions and got this error:

enter image description here

How can I give full control to ALL FILES in this goddamn folder (CLion 2020.3.3)?

I’ve already done this, by the way, but Windows does not acknowledge:

enter image description here

If anyone knows how to fix this issue, please, help!

startup – persistant usb script running on boot (apparmor)

I have ubuntu persistant usb and i have issue with open apps on startup because apparmomr

And i have sollution on startup in terminal i write next:

sudo apparmor_parser -r /etc/apparmor.d/snap-confine

And my question is how to make sort of script for automatic running to solve openning software.

Thanks

i moved to ubuntu and my camera does not work. i was running macOS before

this is the. can any one help me.
lsusb:

Bus 002 Device 003: ID 05ac:8406 Apple, Inc. Card Reader
Bus 002 Device 013: ID 2109:0817 VIA Labs, Inc. USB3.0 Hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 004: ID 05ac:0291 Apple, Inc. Apple Internal Keyboard / Trackpad
Bus 001 Device 010: ID 05ac:828f Apple, Inc. BRCM20702 Hub
Bus 001 Device 003: ID 0a5c:4500 Broadcom Corp. BCM2046B1 USB 2.0 Hub (part of BCM2046 Bluetooth)
Bus 001 Device 022: ID 2109:8817 VIA Labs, Inc. Apple Internal Keyboard / Trackpad
Bus 001 Device 021: ID 046d:c092 Logitech, Inc. BRCM20702 Hub
Bus 001 Device 020: ID 046d:c517 Logitech, Inc. LX710 Cordless Desktop Laser
Bus 001 Device 019: ID 2109:2817 VIA Labs, Inc. USB2.0 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

boot – Running Barrier at startup

I’m having trouble running barrier at startup.
I was able to make it run after logging in, the thing is I want to use my main mouse and keyboard from another pc on this one. meaning I either need to keep it always on, or connect a mouse and keyboard to log in.
any idea on how to bypass it to make it run on boot?
thanks!

linux – Host internal port redirection while blocking target port externally and having docker running

This might be simple but after spending ~5 hours w/ iptables and googling, I couldn’t make it work, so I thought I’ll ask here directly.

I have an ubuntu machine with Docker installed (so it manipulates iptables..)
I have a container that runs in it, and listens on port 80

On the host machine I need to both redirect incoming external traffic whose dest_port is 8080 to port 80 (I would prefer this to happen as early as possible, w/o assuming the listener is a docker container / docker-user chain at the iptables), but if that’s not possible I can compromise on assuming it’s a docker that listens on port 80.

And, to drop any traffic incoming into port 80 externally.

I’ve tried:
PREROUTING,
PREROUTING w/ setting a mark on tcp dest port 80, later dropping all marked packets at INPUT chain

the redirection works, external browsing to 8080 works.
the problem is, external access to 80 works as well.

Any ideas?

(I prefer the external filtering to be interface-specific, but that’s not a must).
(I also prefer blocking “all” incoming traffic from that specific interface, but any attempt to switch INPUT to deny/drop make the entire machine’s net non-accessible even internally)

Many thanks in advance !

DreamProxies - Cheapest USA Elite Private Proxies 100 Private Proxies 200 Private Proxies 400 Private Proxies 1000 Private Proxies 2000 Private Proxies ExtraProxies.com - Buy Cheap Private Proxies Buy 50 Private Proxies Buy 100 Private Proxies Buy 200 Private Proxies Buy 500 Private Proxies Buy 1000 Private Proxies Buy 2000 Private Proxies ProxiesLive Proxies-free.com New Proxy Lists Every Day Proxies123