zfs1/svn 0 Mon Mar 13 17:47:22 2017 zfs1/svn 1 Mon Mar 20 11:00:00 2017 zfs1/svn 2 Sun Mar 26 04:00:00 2017
I wrote a simple server to listen on suspect ports like 23 and those used by Mirai and its descendants, just so I can allow a full TCP handshake before banning a network. This helps prevent SYN DoS attacks from spoofed addresses causing me to block unintended networks.
mcblock now has the ability to grok mail logs for suspicious activity. This is wrapped up in a class just like the auth log parser, and later will use the in-development secure network API to my gateway to add addresses to be blocked.
Speaking of which... I've put a little work into mcblockd, a secure service that will allow remote manipulation of pf tables and my database of blocked networks. This will allow me to stop doing some things by hand, like grokking my web and mail server logs and activating blocks on my gateway from the results.
FreeBSD was the only platform on my list where this bug was present. Hopefully someone will commit my patch or an equivalent patch to fix the issue. I'm a happier camper when the unit tests for libDwm don't fail due to an OS library bug without an easy workaround. The bug I submitted is 207918.
I also wrote more unit tests for libDwm.
I moved these clases and a Gpio class into a separate DwmPi library since I may need them for projects other than my garage door opener accessory at some point.
I created a new RotaryEncoderReader C++ class for FreeBSD, and it works fine. It monitors the A and B channels of the encoder in a thread and can signal another object when the encoder moves clockwise or counterclockwise. Internally it's using a state machine to suppress incorrect transitions. Since the signalling of another object is done in thread context, the receiving object needs to be thread safe. In my case, I will use an instance of my Thread::Queue class template to receive the events. As a bonus, this makes it easy to determine if the encoder has stopped since I can call TimedWaitForNotEmpty() from my Thread::Queue class template. If it returns false, I know that no events were added during the timeout I specified.
So now I'm moving ahead with using FreeBSD on the Raspberry Pi 2 for my garage door opener accessory.
I created a new Dwm::Thread::Queue C++ class template in the Dwm library to use to feed encoder data from one thread to another. This class template is essentially an update of my old Pthread::Queue class from the year 2000, but using standard C++ mutex and condition_variable facilities instead of pthread. I wrote unit tests for it and it appears to work fine. In the process I discovered that I had never updated my UnitAssert framework for multithreading. For now I've fixed it with one mutex with poor granularity. This should be fine for now, my unit tests do not need to be high performance. I ran my unit testsing for my new class template and UnitAssert on my web server and Raspberry Pi 2.
dwm@rpi2:/home/dwm/src/dwm/libDwm/tests% ./TestThreadQueue 6003/6003 passed
The bigger problem at the moment is that there is no emacs package available, which means I can't really do development work directly on the Raspberry Pi with FreeBSD. I'll work on that soon.
The good news for FreeBSD on the Raspberry Pi is that there's a reasonably friendly API for GPIO basics. And given that the RPi2 has a 4-core processor, it's not a huge problem to create a thread that reads the pins connected to the rotary encoder and pushes changes into a queue to be consumed by another thread. The pin-reading thread could use a state machine to avoid sending noise to the consumer.
The other upside to FreeBSD is that I don't have to deal with sysfs stupidity like needing to export/unexport, wait for filesystem entries to appear and disappear, etc. While there are alternatives to sysfs with linux on the Raspberry Pi like PIGPIO, they're not part of the kernel nor included in a base distribution.
Software-wise, today I completed the first pass at a configuration file lexer and parser. I also wrote the unit tests for them.
So far I've spend most of my time on state machines for rotary encoders. It took me a while to find the documentation I needed for GPIO access under Raspbian "jessie" on the Raspberry Pi 2 Model B. And the interface is somewhat clumsy, but I have code working now. I have two state machines: one that handles the gray code from the encoder (using edge interrupts) and a higher-level state machine with states Stopped, Closing and Opening. I've tested it on the Rapsberry Pi 2 I have on hand, and it works.
I'm at about 7,700 lines of code, and still trying to refactor as I go to keep it reasonable. Once I have drang and drop in the tree view, I'll consider the code quite usable for my own purposes. It should easily wind up being less than 10,000 lines of code at that point. It might still be there after I add support for video.
Over the weekend I implemented password changing. I haven't yet added auth token invalidation/recreation when a password is changed, but it should be easy to do. Much easier than all the work I sunk into trying to make the wthttpd connector work in a stable manner.
I also did a little bit of restyling to flatten the interface. I still need to flatten the montage images for albums.
Here's a video taken while importing some of my car photos. This is running as a FastCGI application, hence the upload progress indicator stays at 0%. But at least the import progress widget works correctly, and that's more important to me. Import begins after the files are uploaded.
There are two ways to search the database. The first involves loading the whole database into memory and searching in memory. This is good for avoiding some I/O overhead when performing multiple searches. The second method searches the database file by loading each entry one at a time. Given filesystem cache and the relatively small size of the database file, there isn't a ton of overhead with the second method, and it conserves memory in user space.
Unlike some other programs, I do not intend to store any metadata that is not contained in the JPEG files themselves. This allows me to completely wipe out the database and regenerate it from the JPEG files themselves with no external metadata needed. At the moment I haven't decided where I'll keep things in various EXIF and IPTC data, but I know I don't want to depend on external metadata. Doing things this way provides the highest level of portability for JPEG files; if I want to move my photos elsewhere, or perform a backward-incompatible upgrade of my software, it will be relatively painless.
I have yet to put anything in the UI that involves the database.
Next I'd like to start adding search functionality, for which I need a simple data store. And I want an abstract album view so I can do things like show photos from a certain date regardless of how they're organized into albums. And I still need a tree view of albums.
I'm now at just over 4,000 lines of code.
Below is a video showing me importing 20 photos, creating a new album and moving the new photos into that album.
I'm at about 3,100 lines of code. Still very reasonable, given that quite a bit of the intended functionality is in place.
I fixed an issue with UI updates during file upload that would occasionally cause a crash. When I switched to using multiple threads for import processing, I missed a case where I was updating a variable from more than one thread. It's now fixed.
I added a new album name dialog class, just so I could have a maintainable destructor. I tweaked some of the styling for album and photo selection.
I'm now using WebSockets. Many operations are much faster, which is a good thing.
I now allow albums to be moved or copied. The drag widget shows the number of albums (including subalbums) being moved or copied, as well as the total number of photos being moved or copied. Multiple albums and photos can be selected for copy or move at one time.
I fixed a nit with dragging... in order to never have the drag widget screw up the AlbumBox or ThumbBox by appearing in the layout, its parent is now the album. I could not find any other solution.
Since I run chrooted, I now require that the gallery binary be setuid root, and I change back to the real user ID right after calling chroot().
I have been unable to resolve the massive memory leaks from GraphicsMagick. I will likely create some small utility programs and just call them to rotate and convert photos.
I'm looking at adding video support so I can upload videos from my iPhone. I'll use ffmegthumbnailer to generate thumbnails.
Multi-photo rotations are now multithreaded: one thread per photo. Later I'll probably use a thread pool, but this works fine for me for now and it's much faster than using a single thread when running on a multi-core processor.
I really should make photo uploads multi-threaded, at least in terms of the import process (orientation fix, creation of medium and thumb sized images).
The new code is at 2,636 lines (not counting comments and blank lines):
% mcloc . 309 ./DwmWWWGalleryAlbum.cc 52 ./DwmWWWGalleryAlbum.hh 32 ./DwmWWWGalleryAlbumBox.cc 60 ./DwmWWWGalleryAlbumBox.hh 58 ./DwmWWWGalleryAlbumBoxToolsDialog.cc 26 ./DwmWWWGalleryAlbumBoxToolsDialog.hh 135 ./DwmWWWGalleryApp.cc 29 ./DwmWWWGalleryApp.hh 28 ./DwmWWWGalleryConfig.hh 28 ./DwmWWWGalleryConfigLex.lex 108 ./DwmWWWGalleryConfigParse.yy 11 ./DwmWWWGalleryCopyOrMoveDialog.hh 19 ./DwmWWWGalleryDragPhoto.cc 18 ./DwmWWWGalleryDragPhoto.hh 156 ./DwmWWWGalleryFileUploader.cc 60 ./DwmWWWGalleryFileUploader.hh 21 ./DwmWWWGalleryFolderNameValidator.cc 22 ./DwmWWWGalleryFolderNameValidator.hh 42 ./DwmWWWGalleryImageSelector.cc 18 ./DwmWWWGalleryImageSelector.hh 244 ./DwmWWWGalleryMediumBox.cc 57 ./DwmWWWGalleryMediumBox.hh 202 ./DwmWWWGalleryPhoto.cc 39 ./DwmWWWGalleryPhoto.hh 232 ./DwmWWWGalleryPhotoAlbum.cc 34 ./DwmWWWGalleryPhotoAlbum.hh 31 ./DwmWWWGallerySubAlbumSelector.cc 17 ./DwmWWWGallerySubAlbumSelector.hh 201 ./DwmWWWGalleryThumbBox.cc 79 ./DwmWWWGalleryThumbBox.hh 50 ./DwmWWWGalleryThumbBoxToolsDialog.cc 20 ./DwmWWWGalleryThumbBoxToolsDialog.hh 118 ./DwmWWWGalleryUtilities.cc 18 ./DwmWWWGalleryUtilities.hh 62 ./dwmgallery.cc 2636 TOTAL
In the queue:
I added basic drag-and-drop copying of photos from an album to any visible sub-album in the thumbnail view. Double-clicking a photo enables and disables dragging of that photo. More than one photo can be dragged at once to an album.
I refactored to allow bulk rotation of photos from the thumbnail view. I'll likely do the same for commenting and deleting photos. It will save mouse clicks and keystrokes for the user.
At any rate, the reason I want the internal path navigation is to allow navigation to an album via a URL. Without this functionality, my gallery is a single-path application. When I want to tell someone to look at an album, I don't want to be forced to tell them how to navigate to it from the top level album; I want to just send them the album's URL.
Lesson learned here: think about internal path navigation from the start so it's part of the design from the very beginning.
Next up: the ability to add new albums, and to regenerate an album's montage image.
By the time I'm ready to deploy my gallery, I will be using FreeBSD 10.x on my web server, and will be using WebSockets instead of FastCGI. Hence my playing around is on a local machine without apache. When deploying behind apache, I'll be using the proxy_wstunnel_module.
When linking with shared libraries, I wind up with a huge list of dependencies and a large memory footprint. I built and installed Wt static libraries, and chased down remaining shared libraries so I could link statically. The memory consumption dropped by more than 50%. There appears to be a memory leak in Wt somewhere, I'll figure that out later.
I have file upload working now, but I need to clean up my example so it will be useful later.
I'm also nearly done adding UPS monitoring to sitehealthd. I'll later add front-end code to view UPS status and history (battery charge level, expected runtime on battery, load, input AC voltage, etc.).
In the process of deploying sitetrafficd, I remembered that its packet processing code assumed that the pcap filter being used would only pass TCP packets. I corrected that assumption, it will now work with all IP traffic. I also discovered that GetLocalInterfaces() from libDwm did not work correctly on FreeBSD 9.1 and OS X Mountain Lion. I fixed it for both of these platforms by switching to using getifaddrs(). It is interesting to note that getifaddrs() behaves differently for root and non-root users; non-root users don't get the inet aliases for interfaces while the root user gets them. This is true on both FreeBSD and OS X.
The old user interface was somewhat ugly (was I aiming for a Star Trek color scheme?), and at some point I'll probably change it. However, it is functional, and I had forgotten how useful it can be. It can ping one or more destinations, and it uses TCP packets for transmission and BPF for timing, so it's fairly accurate and not as dramatically affected by process scheduling, etc. as timestamping in application code. Clicking on the plot causes a snapshot to be saved in /tmp; this is useful but also a nuisance since it doesn't let you supply a filename and is easily triggered when you don't want a snapshot.
I used Qt for the GUI, and I've no plans to change to something other than Qt.
The buttons needs some work, but here are some screenshots of what it looks like right now.
This first screenshot is with 2 destinations on my local network: ria and www. Here we have the live plot showing for ria. The packet rate was 1 packet per second. The blue lines and cyan dots in the plot are raw round-trip samples. The yellow line is the median of the last N points, whenre N was 1600 for this picture. If packet loss is oberved, it appears as a red area plot using the Y-axis on the right. This axis is logarithmic because TCP begins to suffer significantly before 10% packet loss.
The next screenshot is the history plot for a single destination on the other side of the country at 20 packets per second. The history plot is a box or candle plot. The top of the blue line represents the 95th percentile of the samples in the box's set. I intentionally don't display the maximum sample; I don't want a single outlier to skew the Y-axis scale. The top of the cyan box represents the 75th percentile. The line through the cyan box represents the 50th percentile (median). The bottom of the cyan box represents the 25th percentile. The bottom of the blue line represents the minimum sample.
The next screenshot is the distribution plot. It shows the round trip time distribution for the last N samples, where N is 1600 in this case.
My web site indexer is working fine and I don't think it needs any tweaks for now. A bunch of things are using its results to get information... search, the menu generator, the sitemap and the ancestry line shown just underneath the site logo.
The first application I have in mind is a very simple web site map.