2026-01-14 Fabio M. Di Nitto Merge pull request #466 from kronosnet/stable1-proposed stable1-proposed 2026-01-13 Fabio M. Di Nitto update copyright to 2026 2025-12-01 Jan Friesse tests: Remove discards const qualifier warnings gcc 15.2.1 started producing warning about "assignment discards ‘const’ qualifier from pointer target type" so patch adds const qualifier to fix this warning. 2025-11-24 Fabio M. Di Nitto build: add support for rust coreutils ... next we drop support for Ubuntu if they don´t upstream their breaking changes ... 2025-10-13 Chrissie Caulfield Annotate coverity reports so that we're not wading in them every time a new version comes out 2025-10-13 Ferenc Wágner Distribute STYLE_GUIDE.md and m4/pkg_check_var.m4 For sake of completeness, mostly, since almost everybody uses Git and pkg-config>=0.28 or pkgconf (where PKG_CHECK_VAR is already included) for development nowadays. 2025-09-25 Fabio M. Di Nitto Merge pull request #446 from kronosnet/stable1-proposed stable1-proposed 2025-09-24 Fabio M. Di Nitto libknet: deprecate SCTP support 2025-09-23 Fabio M. Di Nitto man: align doxyxml with libqb for solaris support 2025-09-23 Christine Caulfield Update README to explain libqb dependancy Several test suite fixes Increase the sleep in pmtud tests Increase timeout in fun_onwire_upgrade Try and make sure IPv6 test addresses are always valid Make knet work on OpenIndiana (Solaris) author: various 2025-09-06 Fabio M. Di Nitto libnozzle - Fix FreeBSD nozzle_down (take 2) also, FreeBSD 13+ now allow tap0 to be delete.. if only the kernel interface was stable.... 2025-09-04 Chrissie Caulfield libnozzle - Fix FreeBSD nozzle_down (#458) For reasons I am unlikely ever to understand, it's necessary to call SIOCGIFFLAGS on a tap interface after SIOCIFDESTROY for the device to actually be removed. ...so do that 2025-08-30 David Hanisch fix wrong tests Api calls fail due to NULL knet handle, but intention is to fail due to wrong arguments. 2025-08-29 Fabio M. Di Nitto libknet: bump soname to reflect new symbol 2025-08-29 David Hanisch Allow to configure a dscp value for KNET_LINK_FLAG_TRAFFICHIPRIO Currently IPTOS_LOWDELAY is used for IP_TOS. A setup may require to set a certain dscp value to prioritize the IP traffic. 2025-08-28 Christine Caulfield Suppress some sudden coverity errors 2025-08-25 Dmitry Belyavskiy Some error checking for OpenSSL functions 2025-08-12 Fabio M. Di Nitto Fix wrong pointer in _handle_recv_from_links_thread() Resolves: https://github.com/kronosnet/kronosnet/issues/449 2025-06-23 Christine Caulfield nozzle: Fix potential thread race in nozzle_close() 2025-06-16 google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> Create and refine STYLE_GUIDE.md for project contributions This commit introduces STYLE_GUIDE.md, a comprehensive document outlining coding standards and best practices for the Kronosnet project. This guide aims to ensure consistency and clarity in contributions. The following key areas are covered: 1. **C Language Style**: * Minimum Dialect: C99. * Indentation: Tabs (not spaces). * Curly Braces: * Control flow (if, for, while): Opening brace on the same line. * Function definitions: Opening brace on the next line. * Always use braces, even for single-line blocks. * Variable Declaration: At the beginning of their scope. * Line Length: Preferred maximum of 120 characters, flexible for readability (e.g., nested structs). 2. **Naming Conventions**: * General: `snake_case` for internal variables and function names. * Public API Elements: * `knet_` or `nozzle_` prefix for functions, structs, typedefs (e.g., `knet_get_host_list`, `nozzle_device_config`). * `KNET_` or `NOZZLE_` prefix for enums and defines (macros), followed by uppercase `SNAKE_CASE` (e.g., `KNET_MAX_CLIENTS`, `NOZZLE_BUFFER_SIZE_DEFAULT`). * Internal Elements: * Names should be descriptive, indicating purpose and subsystem. * Functions shared across multiple threads: Prefix with `_` (e.g., `_shared_data_access`). 3. **Comments**: * `//` for single-line, `/* ... */` for multi-line. * Emphasizes clear and concise comments for non-obvious code. 4. **Best Practices**: * API Changes Require Tests: Any internal or external API modification must be accompanied by corresponding tests in the test suite. 5. **Copyright**: * A copyright notice specific to `STYLE_GUIDE.md` (Year 2025, Author: Jules ) has been included and formatted as plain Markdown text. This initial version of the style guide addresses issue #22 and incorporates iterative feedback on formatting, content, and specificity. A trivial change (extra newline at EOF) was included in this commit to facilitate this comprehensive message. 2025-04-24 Fabio M. Di Nitto Merge pull request #438 from kronosnet/stable1-proposed stable1-proposed 2025-04-22 Fabio M. Di Nitto Update copyright udp: lower log levels triggered for each incoming packets 2025-03-20 Christine Caulfield compress: use lzo1x_decompress_safe lzo1x_decompress() can SEGV if bad data is passed to it. lzo1x_decompress_safe() is slower but ... safer. Issue #435 2024-12-02 Fabio M. Di Nitto Merge pull request #429 from kronosnet/stable1-proposed stable1 proposed Convert time_t to unsigned long long before formatting (#431) Conventionally it's signed long under the GNU C library, which is 32 bits on 32-bit architectures, but Glibc provides the __TIME_BITS==64 option, which makes it signed long long instead (64 bits) and makes no difference on 64-bit architectures [1]. Since the ping_interval and pong_timeout members of struct knet_link are unsigned, we can make this conversion safe by rejecting negative values as well (besides zero). In the end this enables using the uniform %llu conversion specifier across all supported architectures. This got exposed by Debian recently transitioning to 64-bit time_t [2]. [1] https://sourceware.org/glibc/wiki/Y2038ProofnessDesign [2] https://wiki.debian.org/ReleaseGoals/64bit-time Co-authored-by: Ferenc Wágner 2024-12-02 Christine Caulfield Check packets come from the correct interface https://github.com/corosync/corosync/issues/750 2024-06-06 Fabio M. Di Nitto Merge pull request #423 from kronosnet/stable1-proposed Stable1 proposed 2024-05-18 Fabio M. Di Nitto [man] update to latest doxyxml from libqb Fix FORTIFY source detection Problem spotted on FreeBSD 15 where gcc is currently broken. The detection can see that the flags are not working, but still propagating the last attempted value into the build, causing a failure. 2024-01-10 Fabio M. Di Nitto Update Copyright year 2023-11-27 cglosner libknet/tests: fix potential overflow with sprintf 2023-11-24 Fabio M. Di Nitto [man] update to latest doxyxml from libqb Resolves: https://github.com/kronosnet/kronosnet/issues/422 2023-10-16 Fabio M. Di Nitto links: compare apples with apples spotted by build on armhf 2023-09-27 Fabio M. Di Nitto Merge pull request #418 from kronosnet/stable1-proposed stable1 proposed 2023-09-26 Fabio M. Di Nitto links: fix conversion Merge pull request #413 from kronosnet/stable1-proposed stable1 proposed 2023-09-25 Fabio M. Di Nitto rx: allow links to be active faster scenario: node A and node B node A completes ping/pong link up process faster than B node A starts to send data to node B old behavior: node B would unconditionally discard data packets new behavior: node B will check internal link status to see if ping/pong link up process is in progress. if the link is sending pings and receiving pongs, it´s safe to assume that the link will go up (since we are also receiving data from the node A). node B will speed up link up and accept data immediately. This should address a overall init race condition when nodes startup are slighly off by enough milliseconds to cause 2 memeberships to form and see fencing fireworks. links: fix ping interval and pong timeout value checking warn if ping interval is lower than thread timer resolution as it has no effects error if ping timeout is lower than thread timer resolution as it causes links to be highly unstable 2023-09-18 Fabio M. Di Nitto nozzle: don´t leak tun/tap interfaces in test suite the need_tun() test check to verify that the kernel module for tun/tap is loaded, was opening a generic /dev/tun (or /dev/tap) to verify the presence of the kernel module. On linux, closing the fd to the tap/tun device is enough to destroy the device. On BSD the device is set to DOWN on closing the fd, but it´s not destroyed. Stop leaking by gathering ifr ioctl info from the generic interface created by BSD and pass it to the ioctlfd to destroy. 2023-08-14 Jan Friesse [crypto] Try crypt and decrypt on crypto_init Crypt (or decrypt) operation might fail in some cases even when initialization succeeded. This problem is known to happen when FIPS mode is enabled and combination of openssl/aes192 and md5 is used. This was not caught during crypto_init so knet user (corosync) was not aware of non-working crypto so it couldn't handle it - and couldn't send/receive any packets. Solution is to try crypt, decrypt and compare of result buffer right after new crypto_instance is created. If any of these operation fails, whole crypto_init fails, crypto_instance is freed and knet user is notified by receiving non-zero return code. 2023-07-10 Fabio M. Di Nitto Merge pull request #404 from kronosnet/stable1-proposed stable1-proposed build: disable fortify check for test binaries build: autodetect fortify source level automatically build: annocheck disable cf-protection currently reporting some MAYBE false positives in fedora 2023-06-07 Jan Friesse [spec] Migrate to SPDX license Both Fedora and openSUSE now recommends to use SPDX shortname format for License. 2023-05-24 Christine Caulfield nozzle: tests: Fix 'set_down' test on FreeBSD-devel The new FreeBSD adds a LOWER_UP flag to interfaces that indicates whether the driver is signalling L1 up. So we need to exclude that from our grep when checking the higher-level interface status. 2023-04-25 Jan Friesse [tx] Remove expensive memsets Memset of dst_host_ids is relatively expensive because structure is large (128KiB) and it is not really needed because it is always fully set later. 2023-04-03 Christine Caulfield log: Add a TRACE log level This also removes the #ifdef DEBUG' conditionals so that those messages can be enabled without a rebuild. backport of #403 2023-01-04 Fabio M. Di Nitto Merge pull request #381 from kronosnet/stable1-proposed Stable1 proposed 2023-01-02 Fabio M. Di Nitto Update copyright [man] update to latest doxyxml from libqb 2022-12-14 Christine Caulfield udp: Move EPERM to be alongside EMSGSIZE in tx error handler EPERM can be returned (tested Linux 5.19 & FreeBSD 13) when a packet is transmitted that is too large for the i/f MTU or an iptables transmit rule - so it makes sense to treat it like EMSGSIZE and trigger a pMTU run 2022-11-02 Christine Caulfield pmtud: Reset/restart pmtud when a node joins If a new node joins with a 'black hole' that sets the effective MTU to be less than the hardware MTU then it will fail to join. So this patch restarts the pMTU processes each time a new node joins so it has a chance to declare it's real MTU in time. 2022-10-14 Fabio M. Di Nitto Fix update copyright script to better deal with upper/lower cases 2022-08-22 Fabio M. Di Nitto Update link to google drive 2022-08-22 Ferenc Wágner m4/ax_pthread.m4: update to latest upstream version (serial 31) To avoid the warning: $as_echo is obsolete message. 2022-07-27 Jan Friesse configure: Conditionalize AC_PROG_CC_C99 Autoconf 2.70 made AC_PROG_CC_C99 deprecated because AC_PROG_CC is performing same set of tests and sets same variables. To remove warning, conditionalize AC_PROG_CC_C99. 2022-07-22 Chrissie Caulfield admin: Fix a couple of small things (#384) Spotted by the internal CI 2022-06-09 Mingli Yu links.c: Fix build with gcc-12 Fix the build failure with gcc-12 when -Og pass to CFLAGS. | /build/tmp-glibc/work/corei7-64-wrs-linux/kronosnet/1.22-r0/recipe-sysroot/usr/include/bits/string_fortified.h:59:10: error: 'link' may be used uninitialized [-Werror=maybe-uninitialized] | 59 | return __builtin___memset_chk (__dest, __ch, __len, | | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | 60 | __glibc_objsize0 (__dest)); | | ~~~~~~~~~~~~~~~~~~~~~~~~~~ | ../../git/libknet/links.c: In function 'knet_link_set_config': | ../../git/libknet/links.c:108:27: note: 'link' was declared here | 108 | struct knet_link *link; | | ^~~~ | cc1: all warnings being treated as errors 2022-06-02 Ferenc Wágner Fix comment typo: Minumum -> Minimum Fix comment typo: compresion -> compression 2022-05-30 Fabio M. Di Nitto Merge pull request #370 from kronosnet/stable1-proposed stable1 proposed 2022-04-29 Chrissie Caulfield [doc] To Do list moved to projects.clusterlabs.org (#379) 2022-04-01 Jan Friesse [logging] Fix Initializing typo 2022-03-23 Fabio M. Di Nitto [tx] fix epoll event size to cope with extra event on send to link spotted by -Werror=stringop-overflow= new gcc feature 2022-03-21 Fabio M. Di Nitto [tests] deal with new annocheck symlink checks command line has changed between f35 and rawhide, so go for a neutral solution 2022-03-10 Fabio M. Di Nitto [tests] update memcheck exceptions 2022-03-10 Christine Caulfield tests: move all tests to new infrastructure and update copyright years 2021-12-14 Christine Caulfield [UDP] Check for EHOSTUNREACH EHOSTUNREACH seems to have been missed from the list of errors checked at transmit time, so add it. Without this, knet can spin under some conditions. (see issue #373) 2021-11-30 Fabio M. Di Nitto [build] fix flag detections for gcc Resolves: https://github.com/kronosnet/kronosnet/issues/371 2021-11-17 Fabio M. Di Nitto [doc] update links to mailing lists and drop obsolete test suite 2021-11-15 Fabio M. Di Nitto Merge pull request #360 from kronosnet/stable1-proposed Stable1 proposed 2021-11-12 Fabio M. Di Nitto [logging] slow down knet a bit when log socket is busy [crypto] improve logging when failing to decrypt packets 2021-11-08 Fabio M. Di Nitto [udp] use ICMP error messages to trigger faster link down detection this solves a possible race condition when: - node1 is running - node2 very fast - node1 does NOT have enough time to detect that node2 has gone and reset the local seq numbers / buffers - node1 will start rejecting valid packets from node2 There is still a potential minor race condition where app can restart so fast that kernel / network don't have time to generate an ICMP error. This will be addressed using instance id in onwire v2 protocol, as suggested by Jan F. [host] fix dst_seq_num initialization race condition There is a potential race condition where the sender is overloaded, sending data packets before pings can kick in and set the correct dst_seq_num. if this node is starting up (dst_seq_num = 0), it can start rejecing valid packets and get stuck. Set the dst_seq_num to the first seen packet and use that as reference instead. 2021-10-26 Fabio M. Di Nitto [crypto] fix nss crypto buffer boundaries usage this issue was detected by temporary increasing knet header size for debugging purposes, and was generating incorrect crypto data by PK11_CipherOp because the new size was overriding the old data. this issue does not affect onwire_v1 or stable1 as it is. [build] fix debug build vs annocheck parameters FORTIFY_SOURCE requires -O1 or higher 2021-10-19 Khem Raj libknet/tests: Correct include path for poll.h Fixes /usr/include/sys/poll.h:1:2: error: redirec ting incorrect #include to [-Werror,-W#warnings] | #warning redirecting incorrect #include to 2021-10-18 Fabio M. Di Nitto [tests] temporary disable annobin lto check for centos8 failure adding -flto as suggested, triggers other annobin internal failures on centos8. 2021-09-17 Christine Caulfield [build] Add annobin build/check support the annobin check is enabled only if all compiler flags, linker flags and annocheck binary are available. the build will use as many of the hardening options required to pass the annocheck regardless. the check is performed / enabled only with gcc. clang currently suffers from some limitations to automatically detect the annobin plugin, that would increase drastacally the complexity of the build system unnecessarely. implementation based on: - https://bugzilla.redhat.com/show_bug.cgi?id=1961686 - https://developers.redhat.com/blog/2019/02/04/annocheck-examining-the-contents-of-binary-files# - https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/developing_c_and_cpp_applications_in_rhel_8/annobin_toolsets Original idea by Christine Caulfield CentOS Stream CI insists on this but it's generally a 'good thing' 2021-09-16 Fabio M. Di Nitto [nozzle] fix tests on machines that don´t have bash 2021-09-16 Ferenc Wágner nozzle: preserve test result across cleanup nozzle: skip tests if TUN support is not available 2021-08-30 Fabio M. Di Nitto Merge pull request #344 from kronosnet/stable1-proposed stable1-proposed 2021-08-26 Christine Caulfield test: Use a pipe to synchronise wait_for_xxx events The condition variable thing clearly not working for us. 2021-08-26 Fabio M. Di Nitto [tests] make execution of functional testing optionals 2021-08-26 Christine Caulfield send: Fix send_sync() send_sync needs a filter to work - enforce this Clear out some arrays that could have random values 2021-08-26 Fabio M. Di Nitto [transports] remove unused internal ACL API with the new ACL code, this internal API is no longer used [acl] major internal rework ACL internal management was somewhat broken and didn´t really do what was advertised for dynamic links. - fix ACL to be truely per host/link as public API advertise - move ACL checking when host/link are known - add extensive tests - fix various minor bugs that went undetected for a bit - enable access list testings on more tests [acl] make knet_link_add_acl a wrapper to knet_link_insert_acl remove lots of duplicated code 2021-08-09 Fabio M. Di Nitto [host] use correct seq_num to reclaim buffers we cannot use incoming packet seq_num to reclaim buffers as that causes havoc. use latest recorded (highest) packet number instead that follows the reclaiming window [rx] increase defrag buffers to better deal with network jitter 2021-07-02 Fabio M. Di Nitto [man] update doxygen2man to latest from libqb 2021-06-30 Christine Caulfield [doc] Add comments to structure members Removes doxygen warnings at build time Needs the latest doxygen2man to actually print anything in the man pages. Signed-Off-By: Christine Caulfield 2021-06-30 Fabio M. Di Nitto [link] don´t allow mix of dynamic and static links with same source address/port. The configuration doesn´t make sense in itself and confuses ACL. [link] use previously cached value as link info might not be accurate [link] don´t leak resources if link configuration fails 2021-06-25 Fabio M. Di Nitto [tests] suppress more libnss3 errors [spec] cleanup some krosnonetd leftovers 2021-06-25 Christine Caulfield Add comment about log_fd in knet_handle_new() Sanitise sockaddr_storage structs coming in the API Before we used to copy the whole sockaddr_storage from the API into our structs. This meant that any junk left at the end by the caller got into them and could cause trouble with some comparisons. This patch zeros out the extra bits of the sockaddr_storage that are not valid. 2021-06-23 Fabio M. Di Nitto Update copyright around manual backport from master/rust-bindings branches 2021-06-13 Chris Walker Require make utility 2021-06-11 Fabio M. Di Nitto [tx] minor optimizations to avoid recalculating socksize over and over 2021-06-11 Christine Caulfield send 'correct' sockaddr lengths to socket calls FreeBSD-devel insists that the length passed into socket calls matches the expected length of the sockaddr it describes. So, eg, when passing a sockaddr_in, the length must be sizeof(sockaddr_in) rather than sizeof(sockaddr_storage) which we were using Signed-Off-By: Chrissie Caulfield 2021-06-03 Jan Friesse [docs] IRC channel is now on Libera Chat 2021-05-07 Fabio M. Di Nitto [coverity] fix 2 buffer initialization (potential) issues discovered by latest coverity scan update 2021-04-28 Fabio M. Di Nitto Merge pull request #331 from kronosnet/stable1-proposed stable1-proposed 2021-04-27 Christine Caulfield [nozzle] fix nozzle_up() on FreeBSD TBH I'm not sure how this ever worked, as we never seemed to call SIOCIFCREATE, but this also fixes the race when the tests are run -j4 (or higher) 2021-04-26 Christine Caulfield [lib] Don't use -rpath rpath seems, understandably, to annoy some analysis tools. We use it to locate the crypto & compression plugins but libknet also needs to use the plugins from the build-tree when running tests. A simple way to do this is have the plugin directory as part of the knet handle then we can override it in the test suite, which already does 'illegal' accesses to some handle fields. OK, it's not the MOST elegant way of doing this perhaps, but it is simple and effective. 2021-04-19 Christine Caulfield [nozzle] Remove inet_ntoa() call inet_ntoa() is deprecated in favour of inet_ntop() (see man page for inet_ntop()). 2021-04-14 Fabio M. Di Nitto [handle] validate handle in public API - add _is_valid_handle() function to verify if a handle is known, via qb_list* - change all (and only) public API to use _is_valid_handle(). - move library config bits to lib_config.c and keep header bit in internal.h. this is more of a commodity need to simplify accessability of above function to the test suite at link time. 2021-04-09 Fabio M. Di Nitto [global] Update copyright [docs] Update README for BSD section [openssl] port knet top openssl3.0-alpha13 openssl 3.0 is not GA yet and there are still API changes around. it´s currently only available in Debian experimental and while not critical for knet, we continue to keep knet updated and be ready for 3.0 GA. 2021-03-20 Christine Caulfield Add some more uses for event-based tests This gets time for the whole test suite (on my machine) down from 10m25 to 5.56 (make check) and from >120m* to 55m29 (make check-memcheck) * I accidentally hit ^C on these tests after 2 hours so it's probably much longer tests: use callbacks to wait for nodes Rather than just wait for a arbitrary amount of time for nodes to come online while testing, use the knet callback mechanism. This should not only save some time in the running of the tests, but also make them less susceptible to slowness on the CI machines. 2021-03-01 Christine Caulfield tests: Fix some random CI failures. 2021-02-24 Fabio M. Di Nitto Drop travis support 2020-12-24 Fabio M. Di Nitto [tests] hide another internal libnss non recurring memory leak 2020-10-19 Fabio M. Di Nitto Merge pull request #319 from kronosnet/stable1-proposed stable1-proposed 2020-10-12 Fabio M. Di Nitto [tests] add more exceptions to deal with Ubuntu latest rebuilds (clang counterpart) 2020-10-09 Fabio M. Di Nitto [tests] add more exceptions to deal with Ubuntu latest rebuilds 2020-09-21 Fabio M. Di Nitto [cleanup] remove unused infrastructure to send HOSTINFO messages those messages are unused in 1.x and if necessary they will be reintroduced in 2.x 2020-09-18 Fabio M. Di Nitto [sctp] check and warn about internal fd tracker fixes new coverity scan error 2020-09-09 Fabio M. Di Nitto Revert "[heartbeat] don´t send ping if transport is not connected" This reverts commit 2b73aff21f6341cfd00022694c47ff9b47e17a0e. The same code already exists at the top of the function [heartbeat] don´t send ping if transport is not connected this could cause socket buffers to fill up with stale junk and generate unecessary errors. 2020-08-31 Fabio M. Di Nitto [tests] hide internal nss memory leak (non-recurring) 2020-08-26 Fabio M. Di Nitto [rx] fix data stats on RX to match TX model 2020-08-20 Fabio M. Di Nitto [rx] fix per link stats for data packets and cleanup all netutils common functions 2020-08-17 kronosnet CI bot <32407685+knet-ci-bot@users.noreply.github.com> Merge pull request #314 from kronosnet/stable1-proposed stable1-proposed 2020-08-13 Fabio M. Di Nitto [threads] make sure to initialize events spotted on fedora rawhide + valgrind [cleanup] drop no longer used poc-code [cleanup] kill all unsupported kronosnetd code corosync can provide similar capabilities right now 2020-08-12 Fabio M. Di Nitto [cleanup] Move public APIs calls to more suitable location 2020-08-03 Fabio M. Di Nitto [crypto] add support for openssl 3.0 2020-07-14 Fabio M. Di Nitto Merge pull request #312 from kronosnet/stable1-proposed Stable1 proposed f6b11c05112c520ccda3b28e7676acbc6b8a7938 [man] fix type comparison Merge pull request #305 from kronosnet/stable1-proposed Stable1 proposed 2020-07-13 Fabio M. Di Nitto [tests] more ignore list of nss internal stuff [tests] ignore openssl errors triggered when building with clang [tests] fix variable increment logic (detected by clang) 2020-07-09 Fabio M. Di Nitto Update copyright [crypto] add support for live reconfiguration [build] update valgrind memcheck exception file [tests] sets defaults for coverity scan check to match CI [tests] add basic framework to create, join and stop many nodes all nodes will be using loopback interface (lo) at different port offset. [tests] use link_get_enable instead of get_status [tests] remove unnecessary shutdown locking the test should be in charge not to shutdown too much at once 2020-07-09 Jan Friesse [build] Use git-version-gen during specfile build Instead of copying parts of git-version-gen for spec target use git-version-gen directly and parse final version into components (rpmver, alphatag, numcomm) and use them. Main reason is to simplify code a bit (sed scripts are a bit repetitive tho), reuse the code and also allow building of RPM from dist tarball generated from non-tagged commit or dirty git (not very useful). The code relies on fact, that hyphen is never used in tagged release name. [build] Fix fallback handling in git-version-gen When fallback is defined and gitarchive file doesn't exists, use fallback rather than UNKNOWN. 2020-07-03 Fabio M. Di Nitto [man] sync doxygen2man from libqb master this change is not required in master branch 2020-06-10 Fabio M. Di Nitto [openssl] allow use of deprecated HMAC interface with openssl 3.0 Port will be done after rekey feature is complete (too much code conflict atm) 2020-05-13 Fabio M. Di Nitto [tests] strip extra symbol information generated by new nm spotted on debian experimental 2020-04-29 Fabio M. Di Nitto Merge pull request #304 from kronosnet/am_conditional Always define the Automake conditional 2020-04-29 Ferenc Wágner Always define the Automake conditional Otherwise ./configure --disable-man fails with configure: error: conditional "BUILD_DOXYXML" was never defined. Usually this means the macro was only invoked conditionally. 2020-04-23 Fabio M. Di Nitto Merge pull request #294 from kronosnet/stable1-proposed stable1 proposed 2020-04-22 Fabio M. Di Nitto [transport] notify transports that a link is down [sctp] fix ifup/ifdown handling for connected sockets [netutils] move sockaddr_len definition where it belongs [links] fix spacing [sctp] make sure outgoing connections are bound to a specific IP address https://github.com/kronosnet/kronosnet/issues/300 if 2 or more nodes in a knet network have a duplicate IP address, for example a virtbr interface with default config, sctp will fail to build the associations and abort all connections. 2020-03-25 Fabio M. Di Nitto [man] use libqb version of doxygen2man when available and add pkg_check_var compat macro 2020-03-18 Fabio M. Di Nitto [build] make sure to use correct libqb include files across 2020-03-17 Fabio M. Di Nitto [lists] add missing build CFLAGS for test suite [lists] drop internal implementation and use libqb one 2020-03-16 Fabio M. Di Nitto [lists] fix build with recent gcc changes 2020-03-09 Fabio M. Di Nitto [tests] fix return code from wait_for_packet 2020-03-04 Fabio M. Di Nitto Merge pull request #288 from kronosnet/stable1-proposed Stable1 proposed 2020-02-28 Fabio M. Di Nitto [stats] allow knet_link_get_status to operate in readlock context - add per link stats mutex - use per link stats mutex across the board note: some threads need to lock for a slightly longer period of time than strictly necessary to avoid reverse-order locking with other mutexes. [stats] allow knet_handle_get_stats to operate in a readlock context - add global stat mutex lock to protect stats updates - use global stat mutex lock across all the threads - fix up some minor bugs: - update RX crypto stats only when crypto is enabled - update compress and crypto stats in a consistent fashion [rx] kill unused variable [tests] rework test suite link port allocation Logic is to try to configure a link with port X and if it fails, try the next port. This avoids port collisions between services and knet test suite. Please note that the implementation in test-common.c is NOT super clean. There is still some redundant code in there that is left on purpose. There is another branch, not yet merged, that implements functional testing framework that does heavy use of those functions. We will clean test-common.c as we port the functional testing branch and make it ready for merging. For now, this is good enough to have a more stable test suite. [transports] use SO_REUSEADDR only for sctp 2020-02-26 Jan Friesse [man] Enhance prio description of POLICY_PASSIVE Some users found description of POLICY_PASSIVE priority confusing (probably because "priority" word is too overloaded) so add some redundancy to make description unambiguous. 2020-02-21 Christine Caulfield man: Change strcat to strncat Oddly, covscan doesn't compain about the use of strcat, but I'm going to pre-empt it, just in case it decides to. man: Fix covscan reports in doxyxml.c This fixes most of the remaining covscan errors in doxyxml.c. The ones that remain are caused by malloced structures being stored in qb_hashtable_maps. These still cause unfreed memory, because the contents of the maps are never explictly freed, but as they are used until the very end of the program (when the OS will free everything) I'm dubious as to whether it's worth doing it in the code - or whether covscan will work out what's going on anyway. 2020-01-31 Fabio M. Di Nitto Merge pull request #267 from kronosnet/stable1-proposed stable1 proposed [global] Update copyright across the board [sctp] major surgery to use only SCTP events to determine socket status - drop concept of on_connected_epoll to determine if socket is ready or not - provide much better debugging output at all levels - incorporate fix from Xin Long to gather socket status at the right time - deal with a recent kernel change on SCTP socket that broke knet (from rhel7): [net] sctp: allow delivering notifications after receiving SHUTDOWN [rx] use defines to determine RX data types vs random numbers also extend a bit to make ready for SCTP extra return codes 2020-01-31 Christine Caulfield [tx] Don't Clear out msghdr for all transports. When sending a message to multiple links, if one of those links is not connection-oriented then msg_name & msg_namelen would be cleared, thus breaking the send to any subsequent non-connection-oriented links. So now, if we need to clear out msg_name & msg_namelen, we take a copy of the msghdr and edit that instead, [rx] Don't return 512 EOF messages from _recvmmsg If recvmsg() returns 0 for EOF then it's going to do so until the error is rectified or read with getsockopt(). But the _recvmmsg() wrapper keeps reading until the vector is full thus returning a block of 512 EOF messages all of which the caller has to plough through. This patch causes _recvmmsg() to return as soon as it has got the first EOF so the the caller can deal with it in good time and not spin looking at the same thing over and over again. I've also fixed a couple of typos in related comments 2020-01-31 Fabio M. Di Nitto [rx] send reply packets only when transport is connected 2020-01-30 Fabio M. Di Nitto [rx] unify latency values to a capped value to link precision keep the patch simple to avoid API/ABI breakage for now for easy backporting [latency] fix incorrect math that could lead to bad latency calculation Also, document a bit better how latency is calculated 2020-01-30 Christine Caulfield [udp] Better fix for -ENETUNREACH This fix for the ENETUNREACH problem works better than the last one in that it also works with Linux kernels > 5.0.0 (which return -ENETUNREACH) if an interfaces is brought down, and also on FreeBSD which returns ENETDOWN. 2020-01-27 Fabio M. Di Nitto [udp] simplify code (same logic) 2020-01-24 Christine Caulfield [udp] don't make socket spin if a network I/F is down UDP treats ENETUNREACH as a temporary error and just retries, but this causes the TX thread to spin just doing sendto() therefore blocking all other traffic. (To reproduce this try starting corosync with 2 links configured in corosync.conf but only one of them configured to the 'right' address - it will spin in a tight loop and need to be killed with -9) SCTP does not seem to suffer from this. 2020-01-22 Fabio M. Di Nitto [nozzle] use interface name size consistently and drop strncpy in favour of memmove [host] use KNET_MAX_HOST_LEN consistently detected by gcc10 2019-11-20 Fabio M. Di Nitto [tests] mark array as static fixes an odd segfault when running the test on ppc when built with clang 2019-11-04 wferi [handle] fix typo in error log message 2019-10-29 Fabio M. Di Nitto [handle] make sure to unlock config handle on failure [RX] silence defrag buffer expiration debug error when using active-active links, it is simply too noisy and doesn't provide very useful information. [TX] discard too big packets when reading from socketpairs [RX] handle short write to the application properly this change affects only applications that are not using knet generated socketpairs to deliver/receive data to/from knet. If an application uses a fd that is not SOCK_SEQPACKET (basically streaming), we have to handle short writes accordingly, and knet will continue delivering as long as there is progress. The application is responsible to verify that the data packet is complete as the delivery is not guaranteed to be complete. The application can either embed the size of the packet in their data structure or use the socket error notification callback that will be invoked in case of errors or 0 data delivery. [RX] Discard incoming packets if knet cannot reply back. 2019-10-27 Fabio M. Di Nitto [build] fix openssl version detection when not using pkg-config 2019-10-18 Ferenc Wágner [test] append newline to knet_send timeout message 2019-10-16 Fabio M. Di Nitto Merge pull request #259 from kronosnet/stable1-proposed Stable1 proposed [test] add packet verification option to knet_bench 2019-10-15 Fabio M. Di Nitto [PMTUd] invalidate MTU for a link if the value is lower than minimum Under heavy network load and packet loss, calculated MTU can be too small. In that case we need to invalidate the link mtu, that would remove the link from the rotation (and traffic) and would give PMTUd time to get the right MTU in the next round. [test] add ability to knet_bench to specify a fixed packet size for perf test [rx] copy data into the defrag buffer only if we know the size of the frame [host] fix defrag buffers reclaim logic The problem: - let's assume a 2 nodes (A and B) cluster setup - node A sends fragmented packets to node B and there is packet loss on the network. - node B receives all those fragments and attempts to reassemble them. - node A sends packet seq_num X in Y fragments. - node B receives only part of the fragments and stores them in a defrag buf. - packet loss stops. - node A continues to send packets and a seq_num roll-over takes place. - node A sends a new packet seq_num X in Y fragments. - node B gets confused here because the parts of the old packet seq_num X are still stored and the buffer has not been reclaimed. - node B continues to rebuild packet seq_num X with old stale data and new data from after the roll-over. - node B completes reassembling the packet and delivers junk to the application. The solution: Add a much stronger buffer reclaim logic that will apply on each received packet and not only when defrag buffers are needed, as there might be a mix of fragmented and not fragmented packets in-flight. The new logic creates a window of N packets that can be handled at the same time (based on the number of buffers) and clear everything else. Fixes https://github.com/kronosnet/kronosnet/issues/261 [host] rename variables to make it easier to read the code 2019-09-26 Fabio M. Di Nitto [tests] add common function to sleep based on how the test suite is running Address issue while waiting for host to be up and PMTUd first run. 2019-09-25 Ferenc Wágner Fix typo: trasport -> transport tests: skip the SCTP test if SCTP is not supported by the kernel For example, module loading is disabled on Debian build daemons. (In the vein of c5aa1c3343703455b480cef5c173f471e1bb020f.) 2019-09-20 Fabio M. Di Nitto Merge pull request #246 from kronosnet/stable1-proposed Stable1 proposed 2019-09-19 Fabio M. Di Nitto [links] fix memory corryption of link structure the index would overflow the buffer and overwrite data in the link structure. Depending on what was written the cluster could fall apart in many ways, from crashing, to hung. Fixes: https://github.com/kronosnet/kronosnet/issues/255 thanks to the proxmox developers and community for reporting the issue and for all the help reproducing / debugging the problem. 2019-09-13 Fabio M. Di Nitto [tests] give PMTUd more time to redetect MTU Ideal fix would be to use PMTUd callback, but that requires a lot of extra test infrastructure. For now just workaround the problem. [tests] fix ip generation boundaries https://ci.kronosnet.org/job/knet-build-all-voting/1450/knet-build-all-voting=rhel80z-s390x/console and similar, when pid = 255, the secondary IP would hit 256 that is of course invalid. 2019-09-12 Fabio M. Di Nitto [nozzle] fix tapX range on newer FreeBSD 2019-09-10 Fabio M. Di Nitto [pmtud] switch to use async version of dstcache update due to locking context (read vs write) 2019-09-09 Fabio M. Di Nitto [links] stabilize latency calculation when nodes are not responsive The following scenario is more of a corner case than normal, but this change allows to better deal with this situation: 1) 2 nodes cluster (corosync) (node A and node B) 2) kill -stop $(pidof corosync) on node A 3) node B will continue to send ping packets to node A 4) node A is accumulating those ping packets in the kernel network socket 5) wait some seconds and unpause node A 6) node A will start processing the ping packets in the queue and send pong replies to node B 7) node B will see an extreme increase of latency due those "obsoleted" ping/pong packets 8) node B, as latency increases, will take longer and longer to notice that node A is down due to the pong_timeout adjustment for latency (required for initial cluster spike). the solution: 1) Use average latency to calculate pong_timeout_adj vs latency_max. Averate latency will go down again in time, while latency_max is never reset. 2) RX thread will filter out all pong packets that have higher latency than currently configure pong_timeout. This barrier should have been in place even before. this solution reduces the latency spike on node B to a perfectly reasonable level and it will all eventually stabilize over time as latency samples increase and latency will reduce. Please be aware that using a pong_timeout smaller than latency will simply mark the link down now. 2019-09-03 Jan Friesse [handle] Set thread stack size on create Musl libc has small stack size for threads. Knet needs ~300KiB (tested at the time when this patch was created). Glibc seems to use ~8MiB. As a compromise, 1MiB is used. [common] Conditionalize RTLD_DI_ORIGIN RTLD_DI_ORIGIN is used to get absolute path of plugin. It is used only for logging useful info and not strictly needed, so use it only when it is defined (only musl is known to author of the patch) [common] Include correct errno.h sys/errno.h is system-specific path and errno.h should be used instead. 2019-08-27 Jan Friesse [man] Fix priority description of POLICY_PASSIVE ... to match source code. 2019-08-21 Fabio M. Di Nitto [sctp] retry locking in case of failure [tx] clean up channel management code for internal communications the code is still not in use but it's more clear and doesn't trigger memory overrun [nozzle] fix a few coverity errors in the test suite [tx] drop unnecessary usleep when sending to localhost [common] make sure string is null terminated [test] simplify flush log allocate on stack only once and make sure strings are null terminated drop useless read loop since log msg are always smaller than PAGE_SIZE and read are atomic at that level [nozzle] avoid tons of possible buffer overruns [PMTUd] do not double unlock global read lock [coverity] add test targets to run coverity automatically [compress] do not overrun allocated array for compress modules [logging] make sure not to overrun buffers by pre-allocating them [compress] don't leak memory in case of errors during zstd init [nozzle] don't leak memory on error [nozzle] fix negative return detected by coverity scan [sctp] cleanup bugs detected in error paths by coverity scan [acl] avoid forward null deferencing [rx] better error report if we can't resolve hostname / port [common] fix dlopen error handling [tests] fix knet_bench coverity errors [sctp] revalidate fd to make coverity scan happy [handle] make sure that the pmtud buf contains at least knet header size [rx] align data types [sctp] free access list only if the socket is valid [sctp] fix deference after null check [coverity] add .travis.yml to integrate CI with coverity scan [compress] fix #if def around BZIP2 testing [compress] fix a few minor space vs tab and code formatting 2019-08-21 yuan ren [compress]Default compression level use 1. add test casees for a module without default.Using default compression level. 2. Discuss with Fabio, invalid compression level not the knet responsible for, so error logged. But if compress success but dstLen larger than srcLen, defualt compression level will be used, because the request level is not effective. Default compress level use Discuss with Fabio, invalid compression level not the knet responsible for, so error logged. But if compress success but dstLen larger than srcLen, defualt compression level will be used, because the request level is not effective. 2019-08-21 Fabio M. Di Nitto Merge pull request #232 from kronosnet/stable1-proposed stable1-proposed [build] bump soname to indicate new API call [PMTUd] add ability to manually override MTU and disable PMTUd [PMTUd] add dynamic pong timeout when using crypto problem originally reported by proxmox community, users observed that under pressure the MTU would flap back and forth between 2 values due to other node response timeout. implement a dynamic timeout multiplier when using crypto that should solve the problem in a more flexible fashion. When a timeout hits, those new logs will show: [knet]: [info] host: host: 1 (passive) best link: 0 (pri: 0) [knet]: [debug] pmtud: Starting PMTUD for host: 1 link: 0 [knet]: [debug] pmtud: Increasing PMTUd response timeout multiplier to (4) for host 1 link: 0 [knet]: [info] pmtud: PMTUD link change for host: 1 link: 0 from 469 to 65429 [knet]: [debug] pmtud: PMTUD completed for host: 1 link: 0 current link mtu: 65429 [knet]: [info] pmtud: Global data MTU changed to: 65429 [knet]: [debug] pmtud: Starting PMTUD for host: 1 link: 0 [knet]: [debug] pmtud: Increasing PMTUd response timeout multiplier to (8) for host 1 link: 0 [knet]: [debug] pmtud: Increasing PMTUd response timeout multiplier to (16) for host 1 link: 0 [knet]: [debug] pmtud: Increasing PMTUd response timeout multiplier to (32) for host 1 link: 0 [knet]: [debug] pmtud: Increasing PMTUd response timeout multiplier to (64) for host 1 link: 0 [knet]: [debug] pmtud: PMTUD completed for host: 1 link: 0 current link mtu: 65429 [knet]: [debug] pmtud: Starting PMTUD for host: 1 link: 0 [knet]: [debug] pmtud: Increasing PMTUd response timeout multiplier to (128) for host 1 link: 0 [knet]: [debug] pmtud: PMTUD completed for host: 1 link: 0 current link mtu: 65429 and when the latency reduces and it is safe to be more responsive again: [knet]: [debug] pmtud: Starting PMTUD for host: 1 link: 0 [knet]: [debug] pmtud: Decreasing PMTUd response timeout multiplier to (64) for host 1 link: 0 [knet]: [debug] pmtud: PMTUD completed for host: 1 link: 0 current link mtu: 65429 .... testing this patch on normal hosts is a bit challenging tho. Patch was tested by hardcoding a super low timeout here: diff --git a/libknet/threads_pmtud.c b/libknet/threads_pmtud.c index 4f0ba0f..5e2b89b 100644 --- a/libknet/threads_pmtud.c +++ b/libknet/threads_pmtud.c @@ -261,7 +271,8 @@ retry: /* * crypto, under pressure, is a royal PITA */ - pong_timeout_adj_tmp = dst_link->pong_timeout_adj * 2; + //pong_timeout_adj_tmp = dst_link->pong_timeout_adj * dst_link->pmtud_crypto_timeout_multiplier; + pong_timeout_adj_tmp = 30 * dst_link->pmtud_crypto_timeout_multiplier; } else { pong_timeout_adj_tmp = dst_link->pong_timeout_adj; } and using a long running version of api_knet_send_crypto_test with a short PMTUd setfreq (10 sec). [PMTUd] rework the whole math to calculate MTU internal changes: - drop the concept of sec_header_size that was completely wrong and unnecessary - bump crypto API to version 3 due to the above change - clarify the difference between link->proto_overhead and link->status->proto_overhead. We cannot rename the status one as it would also change ABI. - add onwire.c with documentation on the packet format and what various len(s) mean in context. - add 3 new functions to calculate MTUs back and forth and use them around, hopefully with enough clarification on why things are done in a given way. - heavily change thread_pmtud.c to use those new facilities. - fix major calculation issues when using crypto (non-crypto was not affected by the problem). - fix checks around to make sure they match the new math. - fix padding calculation. - add functional PMTUd crypto test this test can take several hours (12+) and should be executed on a controlled environment since it automatically changes loopback MTU to run tests. - fix way the lowest MTU is calculated during a PMTUd run to avoid spurious double notifications. - drop redundant checks. user visible changes: - Global MTU is now calculated properly when using crypto and values will be in general bigger than before due to incorrect padding calculation in the previous implementation. [PMTUd] fix MTU calculation when using crypto and add docs [docs] add knet packet layout [udp] log information about detected kernel MTU [crypto] fix log information [threads] allow knet_handle_setfwd to flush socket queues [tests] ignore libnss errors from OpenSuse Tumbleweed [tests] ignore libnss errors from OpenSuse 15 2019-06-12 Fabio M. Di Nitto Merge pull request #223 from kronosnet/stable1-proposed Stable1 proposed [global] update copyrights [global] clarify license entry per file to match README.licence libraries code: LGPL-2.0+ binaries code and other files: GPL-2.0+ 2019-06-11 Fabio M. Di Nitto [doc] fix a merge oversight from 541d7faf9068d10e12b4278c35825ce1353db081 [crypto] hide errors generated by openssl 1.1.1c see also: https://github.com/kronosnet/kronosnet/issues/226 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=930061#12 [crypto] openssl: drop calls to RAND_seed as they don´t really help RNG See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=930061#12 for reference 2019-06-06 yuan ren [crypto]openssl error strings release In versions prior to OpenSSL 1.1.0, ERR_free_strings() releases any resources created by ERR_load_crypto_strings. 2019-06-03 Fabio M. Di Nitto [transports] fix incorrect merge when cherry-picking 7033ddab505a0cf3655115fe5037579b7c882a8c [crypto] fix openssl1.0 initialization code [PMTUd] extend internal rerun API to allow full PMTUd reset [crypto] rework knet_handle_crypto external API to be more solid the API was rather weak and could potentially leave traffic uncrypted in case of certain, corner case, failures. this patch is a subset of a bigger rework of the crypto layer that will in future allow runtime reconfiguration without traffic disruption of the crypto config. [crypto] make sure to trigger a PMTUd rerun on each good crypto config change [PMTUd] create common/shared code to trigger PMTUd rerun [crypto] make sure to clear all security info on crypto_fini 2019-06-03 Christine Caulfield misc: Fix more covscan warnings The only serious bug here is in transport_udp.c (see bottom of patch), the rest are mostly detail. covscan still reports a lot of errors against doxyxml, most of which are because it doesn't understand the libqb hashtables. 2019-05-20 Christine Caulfield misc: some coverity fixes In rough order of seriousness: 1. Fix clock_gettime() in pmtud so that it's always called, as variable 'clock_now' is always read. 2. Allow space for trailing NUL in libnozzle device names 3. Fix api_nozzle_run_updown_test so it can run out of the build tree 4. Disallow a 0 length prefix in libnozzle 5. Fix potential use of NULL pointer on doxyxml 6. Free 'name' in doxyxml as it's *not* in the map any more 7. Fix dead code in libknet API functions left by code changes 2019-05-17 Fabio M. Di Nitto [spec] use ldconfig_scriptlets only when defined [spec] drop support for init scripts no rpm distros left that support old fashion init scripts [spec] fix a bunch of rpmlint errors [spec] reconciliate fedora spec file into upstream spec file (part 1) [spec] clean up useless conditionals and defines fix a couple of minor conditionals in the process [spec] be more strict about plugins version and architecture depedencies [spec] use distro conditionals to determine BuildRequires [spec] fix upstream URLs to point to https and official release repo Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1708616 also to be noted, the Source0: line is different from upstream and Fedora because upstream can handle tarballs during development 2019-05-09 Fabio M. Di Nitto [build] bump soname to indicate new API calls [global] update copyright across the board [man] fix libknet.h for errors detected by newly added test [tests] improve wait for packet implementation to flush logs during wait [tests] hide an arm internal memory leak (non-recurring) [compress] add support for libzstd 2019-05-09 Chrissie Caulfield manpages: Document enums (#206) And also fix a bug in structure printing that caused it to print the wrong name for a struct. 2019-05-09 Fabio M. Di Nitto [tests] remove stray comment [logging] fix log target of recently added API calls spotted during sctp testing [access lists] add more extensive test for links_acl_ip 2019-05-09 Christine Caulfield acl: Fix English in commments 2019-05-09 Fabio M. Di Nitto [access lists] add public API tests [access lists] improve checks on various data types [access lists] test implicit access lists management for UDP, SCTP and LOOPBACK [access lists] add external API calls to manage access lists [access lists] add documentation for enable_access_list [transports / access list] add internal API to gather which fd to use for access lists given a certain link struct this is required for the external API that has to be transport indepedent [access lists] rename ip1/2 to ss1/2 to keep it more generic [access lists] use arrays to access per-protocol functions [access lists] use better name for fd_tracker structure [access lists] confine access lists data structs within the protocol itself [access lists] add errno around and start using them [access lists] fix build on BSD and add some include files around [access lists] make internal API consistent [links] rename tranport_type to transport to avoid confusion (part 2) complements be9d053efafc822cabd696914d53b5dfe25fb4fd due to early cherry-pick of 7033ddab505a0cf3655115fe5037579b7c882a8c [links] rename transport_type to transport to avoid confusion [access lists] remove 2 unnecessary wrappers [access lists] cleanup API a bit [access lists] more use of generic wrappers and remove duplicate code [access lists] move access lists structs and data types to links_acl.* [access lists] move all acl wrappers to links_acl* and split links_acl_ip to their own files [access lists] fix build on freebsd don't use malloc.h, obsoleted by stdlib.h define s6_addr32 that's only available in kernel space [access lists] add access lists support to sctp [access lists] enable generic access lists only for protocols that use them protocols such as SCTP that use their own access list tracking will need to setup access lists in transport_link_set/clear_config [access lists] enable access lists for GENERIC_ACL protocols (udp for example) [access lists] allow knet_bench to enable/disable access lists [access lists] add tests for default access lists [access lists] automatically add and remove point to point access lists those are not used just yet. [handle] properly initialize fd tracker buffers [access lists] make code more generic to accept more than IP protocol and start to bind it to each fd access lists are unique per file descriptor, each fd can have its own protocol and list. remane around ipcheck* with check* to be more generic. [transports] add information about the nature of the transport and supported access lists [acl] add knet_handle_enable_access_lists api call [acl] move poc-code into libknet dir and rename to links_acl.* code is not integrated yet and test suite can´t run standalone 2019-05-09 kronosnet CI bot <32407685+knet-ci-bot@users.noreply.github.com> Merge pull request #211 from kronosnet/stable1-proposed stable1-proposed 2019-05-02 Fabio M. Di Nitto [udp] improve error message decoding from ICMP errors [udp] use defines vs hardcoded numbers 2019-05-01 Fabio M. Di Nitto [man] fix libknet.h for errors detected by newly added test 2019-04-30 Fabio M. Di Nitto [tests] add man page check to verify doxy header order and definitions [man] fix knet_host_set_policy parameters order 2019-04-30 Christine Caulfield man: Tidy more man pages Followup to previous 'tidy' 2019-04-30 Chrissie Caulfield man: Tidy manpages (#215) * man: Tidy manpages for libnozzle doxygen works in mysterious ways, adding a blank line before @brief makes the lines following that much tidier. So now instead of nozzle_close nozzle - pointer to the nozzle struct to destroy we get: nozzle_close nozzle - pointer to the nozzle struct to destroy * doxyxml: Cope with pointers-to-pointers passed as params Double pointers showed as ' * *name' when they should be ' **name'. Also tidy STRUCTURES display so that they are not indented too much, * man: Similar @brief fixes for libknet.h * doxyxml: Tidy descriptions of functions as parameters If a complex function pointer was passed as a parameter then doxyxml tryied to line up all the other parameters with it - making a mess by having lots of blank space between the type and the name. Now we enforce a maximum type length (a line-ish) so that shorter tyopes will line up OK and the really long ones will be left to their own devices. 2019-04-11 Fabio M. Di Nitto [crypto] remove libnss 3des support 2019-04-09 Fabian Grünbichler reduce minimum crypto key size to 1024bit Since the key is used for AES/3DES and HMAC operations only, this is safe. AES/3DES use keys in the 128- to 256-bit range, HMAC with MD5/SHA1/SHA2 should use keys with a minimum of 128- to 512-bit (in both cases, depending on the actual algorithm used). This reduction also keeps knet compatible with existing Corosync 2.x keyfiles, which are 1024-bit. 2019-03-27 Fabio M. Di Nitto [build] add another exception to valgrind nss combo Merge pull request #199 from kronosnet/stable1-proposed stable1-proposed 2019-03-26 Fabio M. Di Nitto [global] update copyright across the board 2019-03-26 Christine Caulfield link: Check address families on a link always match We can't create a link with a source address of a different address family than the destination as all sends will fail. 2019-03-26 Fabio M. Di Nitto [transports] fix usage after free of transport info memory spotted while implementing functional testing for rekey. when configuring multiple hosts (3+) with 1 link sharing the same listener, but NOT enabling the link, if the user attempts to shutdown knet_h, the first call to knet_link_clear_config will not recognize that the listener is still in use by another link and will release all resources. At the time of invoking clear_config on the second link, transport_clear_config will access already freed memory, or alternatively, a call to link_enable would cause a crash because the listener does no longer exist. [tests] test rpm should match currently installed version of libknet 2019-03-12 Chrissie Caulfield manpages: Document enums (#206) And also fix a bug in structure printing that caused it to print the wrong name for a struct. 2019-03-11 Fabio M. Di Nitto [tests] allow shipping of the test suite [host] increase log level for currently used links bug reported on IRC by yuanren [transport] don´t specify destination address for connection oriented protocols some implementations of sendto/sendmsg expects dst_addr / msg_name to be empty for connection oriented protocols such as SCTP (spotted on FreeBSD). Make sure to clear or not pass those information on connection oriented protocol. 2019-03-10 Fabio M. Di Nitto [man] fix man page build dependency for BSD Make [tests] add ability to create ipv6 sockets [man] fix libnozzle man page generation target [tests] fix knet_bench help text 2019-02-28 Christine Caulfield doxyxml: Don't crash if there's no doxygen stanza for a function 2019-02-28 Fabio M. Di Nitto [man] add test to make sure all generated man pages are being shipped / installed 2019-01-30 kronosnet CI bot <32407685+knet-ci-bot@users.noreply.github.com> Merge pull request #183 from kronosnet/stable1-proposed Stable1 proposed 2019-01-24 Jan Pokorný [common] fix not reporting a true dlinfo error cause This was automatically caught with GCC 9 ("'%s' directive argument is null"). 2019-01-07 Fabio M. Di Nitto [libknet] allow better handling of internal threads while playing around adding some dynamic internal threads, i did hit a limitation with allocating a dynamic THREAD_MAX value based on OS availability of certain features. this patch changes from dynamic to static allocation of threads, that can always be increased if necessary without breaking anything. simplify thread numbering and conditionals on OS features improve debugging of thread status by adding a 2 phase thread registration process: - register a thread before starting it with THREAD_REGISTERED before pthread_create - set a thread status as started with THREAD_STARTED any failure in between can now be detected in debugging logs. Note for developers: all threads *MUST* be registered/started before wait_all_threads_status check in handle.c. 2019-01-04 Fabio M. Di Nitto [build] enable all compilers warnings at once 2019-01-03 Fabio M. Di Nitto [libknet] expose API version in libknet.h to make it easier for applications to add conditional builds 2018-12-28 Ferenc Wágner Skip SCTP event initialization if the protocol is not supported We may not want to actually use SCTP after all, and if we do, we won't get to use the uninitialized subscribe struct anyway. Thus aborting handle initialization is premature here. Fix nozzle man dates to SOURCE_EPOCH, just like we fix the libknet ones 2018-12-19 Fabio M. Di Nitto [rpm] ship libnozzle man pages 2018-12-19 Ferenc Wágner Fix typos Install the nozzle man pages 2018-12-19 kronosnet CI bot <32407685+knet-ci-bot@users.noreply.github.com> Merge pull request #162 from kronosnet/stable1-proposed stable1 proposed 2018-12-18 Fabio M. Di Nitto [knet] fix build with older openssl versions spotted on freebsd-devel [global] update notes to package maintainers 2018-12-18 Ferenc Wágner Tabify Use the kernel struct size for SCTP event subscription 2018-12-18 Fabio M. Di Nitto [libnozzle] fix variable assignment order [global] fix copyright across [libnozzle] add nozzle_get_ips tests and kill nozzle_test [libnozzle] add nozzle_del_ip tests [libnozzle] fix del_ip removal of Ipv6 addresses when MTU < 1280 [libnozzle] add nozzle_add_ip tests [libnozzle] add api_nozzle_run_updown tests and cleanup lots of craft around related to the old testing method for the same API [libnozzle] add internal execute_bin_sh_command test [libnozzle] cleanup tests Makefile.am [libnozzle] add nozzle_get_fd test [libnozzle] add nozzle_get_name_by_handle test [libnozzle] add nozzle_get_handle_by_name tests [libnozzle] add nozzle_get_mac / nozzle_set_mac / nozzle_reset_mac tests [libnozzle] add nozzle_get_mtu / nozzle_set_mtu / nozzle_reset_mtu tests [libnozzle] move multi nozzle device test to api_nozzle_open [libnozzle] clean up code in set_up test [libnozzle] improve set_down test [libnozzle] add api_nozzle_set_(up|down) tests [libnozzle] drop unnecessary indirection to call set_down [libnozzle] drop redundant check [libnozzle] add api_nozzle_close_test also, share more code in test-common, fix another BSD build issue [libnozzle] fix test suite build on BSD [libnozzle] add api_nozzle_open test [libnozzle] prepare Makefile.am to explode with API tests also fix nozzle_test for using global compiler flags [libnozzle] fix API check paths [libnozzle] add per-API call test coverage check [libnozzle] re-instate error checking in nozzle_get_ips [libnozzle] add comment about BSD ip_add status [libnozzle] cleanup nozzle_get_ips public API [libnozzle] fix whitespaces [global] Update copyright across [libnozzle] Update copyright [libnozzle] fix build on BSD [libnozzle] remove error_string requirement from many API calls and mark them as final [libtap] move nozzle_run_updown to completed section [tests] properly check symbols on PPC64 [tests] remove debugging sleep() [libnozzle] enable libnl3 < 3.3 build workaround fixes build on opensuse-42 [libnozzle] Use libnl3.0 for ip address management on Linux BSD will follow next using internal ioctls and drop usage of forking a shell [libnozzle] whitespace cleanup [nozzle] expand buffers to deal with strncpy and -Werror=stringop-truncatio [nozzle] drop duplicated code [nozzle] enable libnozzle build by default [nozzle] move man pages to top level dir [nozzle] fix man page generation [nozzle] propagate errors all the way [nozzle] fix build on BSD [nozzle] nozzle_open and nozzle_close should be stable now [nozzle] move set up and down to the stable API section [nozzle] decouple running pre-up.d/up.d/down.d/post-down.d from interface status (part 4) [nozzle] decouple running pre-up.d/up.d/down.d/post-down.d from interface status (part 3) [nozzle] decouple running pre-up.d/up.d/down.d/post-down.d from interface status (part 2) [nozzle] decouple running pre-up.d/up.d/down.d/post-down.d from interface status (part 1) provide the facility to do it via nozzle_run_updown but delegate the task to the application. This has the benefit of much better fine grained control over errors during those code paths. [nozzle] move find_ip to internals [nozzle] move ipv4 broadcast to internals [nozzle] cleanup mac address functions [nozzle] move code around to reflect public API and internal functions cleanup _get_mtu in the process [nozzle] move code around to reflect completion and drop copy of code from internals.c [nozzle] rename _check to is_valid_nozzle, make it more useful and fixup error code checking [nozzle] be more explicit on function naming (part 2) [nozzle] be more explicit on function naming [nozzle] start cleaning internal function names [nozzle] split code around to separate libnozzle and testsuite [build] split tests, man pages and build dirs [nozzle] rename lib_mutex to config_mutex [build] fix build on BSD [cleanup] document nozzle_iface struct, drop unnecessary ifr struct and cleanup names [cleanup] rename ip_addr to ipaddr [cleanup] rename sockfd to ioctlfd [cleanup] rename struct _ip to struct nozzle_ip and cleanup related defines [nozzle] rename _config to nozzle_lib_config [cleanup] start splitting the code across multiple files [nozzle] cleanup and document nozzle_get_fd [nozzle] cleanup and document nozzle_get_name_by_handle [nozzle] cleanup and document nozzle_get_handle_by_name [nozzle] cleanup and document nozzle_(re)set_mac [nozzle] cleanup and document nozzle_get_mac 2018-12-18 Christine Caulfield Tidy some English 2018-12-18 Fabio M. Di Nitto [nozzle] cleanup and document nozzle_set_mtu and nozzle_reset_mtu couldn't split those 2 because they are stricly related [nozzle] cleanup and document nozzle_get_mtu [nozzle] fix test on BSD [nozzle] cleanup and document nozzle_get_ips (part 1) [nozzle] cleanup and document nozzle_del_ip [nozzle] cleanup and document nozzle_add_ip [nozzle] re-order functions in order of importance [nozzle] cleanup and document nozzle_set_down [nozzle] cleanup and document nozzle_set_up [nozzle] start to cleanup nozzle_close to be more informative and effective [nozzle] cleanup nozzle_open coding style and return codes also update the header file for Doxygen man pages [nozzle] rename last tap bits to nozzle [nozzle] rename API from tap_ to nozzle_ [nozzle] fix pkg-config to point to nozzle [nozzle] rename struct _iface to nozzle_iface (looks better on public headers) [nozzle] rename tap_t to nozzle_t [nozzle] include net/if.h to define IFNAMSIZ instead of redefine [nozzle] add Doxygen header [libnozzle] rename libtap to libnozzle (part 1) Our fearless Debian maintainer wferi made us aware that TAP is an overloaded term already and it would be very confusing to add yet another one. nozzle is a good pick from http://www.thesaurus.com/browse/tap I am sure lib*cock* would have cause havoc :-) This commit only rename files and allow builds. The API needs to be renamed too, but that will come over time with major code cleanup as documented here: https://trello.com/c/pyEKd4UH/396-libnozzle-make-it-production-ga-ready [libtap] BSD: switch to ioctl to destroy tap devices [libtap] define IFNAMSIZ for applications that don't need/want to include net/if.h [libtap] update README for BSD [libtap] port to freebsd and fix a bunch of small bugs - tap_open: do mutex lock only after malloc to avoid deadlock - fix test suite to increase randomness when testing named tap devices - remove hardcoded paths to /bin/true /bin/false and /bin/grep [libtap] tests: add better IPv6 private address space randomization [libtap] tests: allow random localhost IPs for testing - avoids clashing with machine routing - allows parallel testing 2018-12-13 Christine Caulfield UDP: Cope with EPERM returned from sendmsg/sendto IPtables can cause -EPERM to be returned from socket sends, so we need to be able to handle it without going into a spin. SCTP seems to be unaffected. 2018-12-03 Jan Pokorný [misc] fix a few typos/stylistics 2018-11-27 Chen Jingpiao [tests] update error message in api_knet_{addrtostr,strtoaddr}.c [tests] fix check condition in api_knet_{addrtostr,strtoaddr}.c clear the errno in each exported API when the call success 2018-11-20 Fabio M. Di Nitto [openssl] fix return type for pthread_self spotted on freebsd-devel 2018-11-16 Chen Jingpiao [tx] fix tx_uncompressed_packets stats collect in _parse_recv_from_sock 2018-11-13 Chen Jingpiao [rx] fix incorrect packet length in _parse_recv_from_links [tx] remove goto statement to improve readability Add static qualifier to {compress, crypto, transport}_modules_cmds 2018-11-12 Chen Jingpiao [transports] fix transport_modules_cmd array out of range Fixes: 18ab4888 2018-11-07 Fabio M. Di Nitto [build] fix supported compiler warning detection move from AC_PREPROC_IFELSE (strongly discouraged) to AC_COMPILE_IFELSE our detection system was very weak and recent versions of clang did show that PREPROC_IFELFE (cpp) would enable warning options that the compiler does not support (clang). use a full compilation test to detect what works and what doesn't. Also expand the warning list to include new / renamed clang options of equivalents already enabled for older versions of clang and gcc. 2018-11-07 Chen Jingpiao tests: fix output message Fixes: 04537c57 ("[transports] cleanup API calls") fix typo man: update XML directory in doxyxml.c Fixes: 31945b18 ("[man] move libknet man pages and man pages build tools to top level dir") 2018-10-23 Christine Caulfield misc: keep covscan happy covscan complains about is ignoring returns from pthread_mutex_lock() but in those 2 circumstances we don't have much choice. This patch just keeps coverity quiet so we don't miss any real bugs it might find. 2018-10-22 Fabio M. Di Nitto Merge pull request #155 from kronosnet/stable1-proposed Stable1 proposed 2018-10-21 Christine Caulfield links: Don't close loopback link knet_link_clear_config would incorrectly clear the local loopback link if the last 'real' link to an external node was closed. We didn't check the nodeid in the code, just the link number when clearing the loopback flag. 2018-10-08 Jan Friesse Import key with length not dividable by wrap key block size Wrapping of the key is standard crypto operation which needs data aligned to cipher block size, otherwise it fails. Possible solution is to use a zero filled buffer with size aligned to required wrap key block size. Private key is copied to the beginning of the buffer and unwrap operation keeps using only required private key size. 2018-09-11 Fabio M. Di Nitto [tests] clear errno to avoid incorrect propagation from void fn call [tests] fix select(2) handling (mostly affects slow machines) 2018-09-06 Fabio M. Di Nitto [threads] make it easier to debug thread startup / stop operations with pretty names 2018-09-05 Jan Friesse Use gitarchive-version for spec file git-version-gen fail on UNKNOWN version Do not match lightweight tags in git-version-gen Fix leftover from 175b517fd71d999150aa701ec847510490a18a6e which added matching of lightweight git tags. Support for git archive stored tags Attempt to solve problem with git archive generated tarballs (used for example by github when release is downloaded) which are no longer git tree and (in contrast to officially released tarballs) also doesn't contain .tarball-version file so git-version-gen script simply cannot obtain valid version info. Solution is based on using gitattributes which is instructs git to replace string in the .gitarchivever file by known ref names. git-version-gen is enhanced to support this file and tries to parse any string which looks like "tag: v[0-9][^,)]*". If such string is found it's used as a version. This file is used as a last attempt and other methods (.tarball-version, git abbrev) have precedence. Based on idea stated by Jan Pokorný . 2018-08-16 Fabio M. Di Nitto [tests] drop arm exceptions as they have been fixed in glibc [tests] cleanup nss memcheck supports add one more exception for newer nss (spotted on fedora rawhide) drop FreeBSD excetions as we don't support running under valgrind on BSD 2018-08-07 Fabio M. Di Nitto Merge pull request #150 from kronosnet/stable1-proposed stable1-proposed 2018-08-03 Fabio M. Di Nitto [threads] use internal array to track status of threads (running/stopped) vs a random sleep() [tests] don't redefine symbols from the main library [tests] adds more overrides for fedora28/arm/glibc [handle] reduce usage of handle_config_mutex to wrap only global lib init [logging] report error in case logging buffer is bigger than PIPE_BUF See https://github.com/kronosnet/kronosnet/pull/148 [handle] drop init_lock_done as it was used only by locked version of log_msg [logging] drop locking inside log_msg (part 1) looking at the code, there are only few calls to log_msg happening outside of locking context (error reports when failing to grab the lock). Everything else appears to be safe. drop the attempt to lock inside log_msg for now, part 2 will address those few calls that can happen outside of locking context. [handle] drop call to log_msg that would have no effect logging is not setup yet at this stage and we can't log just yet. 2018-07-17 Fabio M. Di Nitto [tests] ignore nss internal memory leak (3.38+) non recurring [tests] reference correct library in memcheck comments 2018-06-08 Fabio M. Di Nitto [tests] add another lzma exception spotted on Debian Experimental, the latest toolchain appears to add full path to objects and valgrind would generate an error. add a generic catch-all path exception. 2018-04-26 Ferenc Wágner coding style: add parentheses after the sizeof operator log_msg: no point in putting NL after the final NUL knet_log_msg: fix documentation: the closing newline is added by flush_logs _logthread: while loop instead of goto _logthread: select() returns the number of set bits, as an int flush_logs: don't ever print past the message buffer flush_logs: while loop instead of goto flush_logs: a for loop is more straightforward here flush_logs: don't repeat the type of msg flush_logs: the reads will overwrite the entire msg structure flush_logs: remove redundant condition The read loop is always entered, resetting len and returning from the function if it isn't greater than 0. 2018-04-23 Fabio M. Di Nitto Merge pull request #145 from kronosnet/stable1-proposed stable1-proposed 2018-04-20 Fabio M. Di Nitto [crypto] NSS_NoDB_Init: the parameter is reserved, must be NULL Thanks to Feri for spotting it in corosync [crypto] improve checks around RX packet size 2018-04-19 Fabio M. Di Nitto Merge pull request #140 from kronosnet/stable1-proposed stable1-proposed 2018-03-27 Ferenc Wágner build: look for source_epoch in srcdir build: check for SOURCE_DATE_EPOCH before turning to Git for an epoch 2018-03-27 Fabio M. Di Nitto [build] fail and provide proper error if source tree version is unknown 2018-03-27 Ferenc Wágner build: git -C is a relatively new option, avoid it build: accomodate to the actual date utility spec: use the source epoch here as well man: use new doxyxml date options to build man pages Based on Fabio's original patch. 2018-03-27 Christine Caulfield man: Add date & year options to doxyxml 2018-03-27 Ferenc Wágner build: distribute the source_epoch file build: determine SOURCE_EPOCH during configuration 2018-02-26 Fabio M. Di Nitto [compress] extend check to compare end buffer data [compress] update valgrind exception file to match new lzma entry points [compress] delegate compress_level validation to compress libraries - make val_level internal api call optional - keep lzo2 val_level around due to the specific nature of compress_level values - add internal compress_lib_test to do a round-robin (compress/decompress) check with provided values - drop unnecessary val_level checks around - update internal compress API include file - adjust api_knet_handle_compress test to use a known bad value for zlib validation [tx/rx] improve error checking when running external code libknet cannot guarantee that the dst_host_filter is sane nor that it cannot be exploited to return garbage to knet. there is an infinitesimal possibility that, if the plugin returns total crap, knet could crash by trying to access out-of-bound memory. Prevent that by adding specific checks both in TX/RX. 2018-02-25 Fabio M. Di Nitto Merge pull request #116 from kronosnet/stable1-proposed stable1-proposed fixes 2018-02-23 Fabio M. Di Nitto [build] fix ChangeLog generation in release tarball 2018-02-20 Ferenc Wágner Indicate new API call in libtool version info build: provide -version-info directly instead of -version-number The info components carry the real OS-independent library version information, from which the version number is computed differently on different systems. According to the libtool manual "new projects should use the -version-info flag instead". New API function: knet_handle_new_ex() taking handle flags For now, the only supported flag is KNET_HANDLE_FLAG_PRIVILEGED, which is also the default in knet_handle_new(). The tests are adapted to switch this flag off, which makes them pass in default environments without privileges. 2018-02-20 Christine Caulfield UDP: Don't keep retrying send if we get EINVAL UDP can return EINVAL if the route is down or not configured, this can cause knet to 'spin' (well, ish, there is a usleep in there). With this patch in place knet doesn't hog the CPU anywhere near so much (and it does recover if the link returns). 2018-02-19 Christine Caulfield doxyxml: Use overridden XML_DIR for reading structures 2018-02-19 Fabio M. Di Nitto [man] fix path to stamp file [man] move kronosnetd man pages [man] move libknet man pages and man pages build tools to top level dir 2018-02-18 Fabio M. Di Nitto [build] simplify Doxygen test based on Feri's input [build] be consistent [build] make man page build optional [build] check for all *int types as we use pretty much all of them [build] drop redundant check (it's duplicated just a few lines below) [build] drop unnecessary function checks [build] drop unnecessary include checks [build] fail configure if we can't find Doxygen to build man pages [build] drop obsolete AC_PROG_LIBTOOL included in LT_INIT [build] enable error when checking --enable-new-dtags linker flags 2018-02-13 Ferenc Wágner build: no binaries in build-aux wander out of the build tree build: better separation of flags during cross-compilation build: kronosnetd still needs libqb on the host system build: compile and link doxyxml for the build host build: use build architecture libraries for doxyxml 2018-02-12 Ferenc Wágner Better wording suggested by Chrissie Reword the documentation of knet_handle_compress_cfg We support several compression libraries by now. 2018-02-03 Fabio M. Di Nitto [compress] lz4: fix builds for older lz4 releases [compress] lz4: include all required headers upon reading lz4hc.h, lz4.h should always be included as depedency 2018-02-03 Ferenc Wágner doxyfile: description of return values may be missing 2018-02-03 Fabio M. Di Nitto [man] drop dist_man_MANS leftover 2018-02-01 Fabio M. Di Nitto [build] make gcc8 buffer boundaries check happy 2018-01-30 Ferenc Wágner doxyxml: use AM_CFLAGS for debugging info, optimization and warnings The rework of the FLAGS handling and the development of doxyxml happened on different temporary branches, and this slipped through on the merge. tests: skip the SCTP test if SCTP is not supported by the kernel For example, module loading is disabled on Debian build daemons. build: --export-dynamic is needed for dlopened modules only Not that it matters on ELF platforms the least. If Kronosnet is ever ported to Windows, however... build: linking with libm is unnecessary on some systems (like OS X) Inlining of builtin functions can eliminate the libm dependency on other systems as well, but AC_SEARCH_LIBS can't detect this because it tests with a false prototype. Thus we use --as-needed linking already to trim such unnecessary dependencies. build: use --as-needed to avoid unnecessary dependency on libm I removed --as-needed in 1b96703, but now I found a good reason to use it after all. build: check for the --enable-new-dtags flag before using it build: sanitize CFLAGS handling As for LDFLAGS previously: leave the CFLAGS "user variable" untouched to enable easy selective overriding of the flags used by the build system. Using $lt_prog_compiler_pic shouldn't be necessary, let's drop it and see if anything breaks. WERROR_CFLAGS was undefined, so dropped as well. build: we preserve CPPFLAGS, just remove the tempting hooks build: sanitize LDFLAGS handling The Automake manual states that according to the GNU Coding Standards the so-called "Variables reserved for the user" mustn't be changed by the build system, and they must override the default settings from the build system. The "Flag Variables Ordering" section provides the recipes on which we build here. Using $lt_prog_compiler_pic directly shouldn't be necessary, let's try leaving it out. build: LIBS belong to LDADD for binaries build: remove unused DEBUG Automake conditional Propagate NSS initialization errors EVP_CIPHER_block_size returns size_t, which is unsigned Propagate OpenSSL initialization errors Add some missing error propagation 2018-01-29 Fabio M. Di Nitto [man] do not ship precompiled doxyxml [man] add build requires on doxygen [man] fix man page generation external deps on doxyxml [man] fix distcheck, build out of tree and tarball generation [man] update .gitignore [man] switch to autogenerated man pages 2018-01-29 Christine Caulfield doxyxml: Fix parameter ordering Also tidy pointer printing and remove extraneous quotes from the BITS IN CAPS (not sure why I did that, but it helps the diff from the old pages) 2018-01-29 Fabio M. Di Nitto [man] don't output unnecessary .PP sections [man] fix comma separated list in See Also section [man] reduce diff with the old man pages O...C...D... [man] first attempt to plug doxyxml to replace doxy2man [man] fix args parsing and help text Update .gitignore [man] init variable and silence gcc [man] fix casting around [man] define _DEFAULT_SOURCE for new features.h [man] add libqb and libxml2 devel as general buildrequires for man pages 2018-01-29 Christine Caulfield doxyxml: Fix compiler warnings 2018-01-29 Fabio M. Di Nitto [man] add makefile.am [man] move doxyman to build-aux [man] drop redundant update-man-page code 2018-01-29 Christine Caulfield manpages: Generate man pages using our own command This removes the dependncy on doxy2man which is not packaged. 2018-01-19 Fabio M. Di Nitto Use LD_LIBRARY_PATH to find the build tree artifacts and disable RPATH across the project The libtool wrapper scripts of the test binaries set LD_LIBRARY_PATH to find the libknet library and its modules in the build tree, but RPATH overrides this setting. This is why RPATH is deprecated, so we switch to RUNPATH instead by using --enable-new-dtags, which is already the linker default on Debian for example. Based on the original patch from Ferenc. 2018-01-18 Ferenc Wágner tests: check for exported symbols in the data section as well The PowerPC64 ELFv1 ABI puts the function descriptor symbols there, but why restrict the consistency check to the text section anyway? 2018-01-17 Bin Liu typo fix: change the url in spec file 2018-01-08 Fabio M. Di Nitto Merge pull request #112 from kronosnet/man-pages [docs] Update copyright and man pages pre-1.0 [docs] Update copyright and man pages pre-1.0 Merge pull request #111 from kronosnet/fixes Fixes [PMTUd] drop (now) unnecessary and dangerous usleep prior to all threads being able to notify PMTUd of EMSGSIZE errors, we had this random usleep in there to have time to collect data. It was working at the time, but it's a bad idea. On super large clusters (>66 nodes) with 4 links on each node, when applying heavy load (cpghum on all nodes at once), the average latency between nodes can increase so much that the PMTUd thread usleep could literally block corosync for seconds at a time. Drop the usleep and live happily ever after [PMTUd] reset value or suffer the pains of DEADLOCKING! errno is per thread, not per function [PMTUd] only notify PMTUd when pmtud_running and don't force a run to avoid lock fighting When we request a writelock, we should notify PMTUd to abort and cond_signal only if PMTUd is running. Also, don't request an immediate rerun as we cannot guarantee that the scheduling is favourable to our request and we could end up waiting forever. [stats] show loopback links as connected 2018-01-05 Fabio M. Di Nitto Merge pull request #105 from kronosnet/scale Scale 2018-01-04 Fabio M. Di Nitto [PMTUd] more typo fixes meeeehhhhhh fixed on the top comment, forgot the bottom one [PMTUd] cleanup variable usage and code around it 2018-01-04 Christine Caulfield fix missing space 2018-01-03 Fabio M. Di Nitto [PMTUd] amend warning message [PMTUd] generate warning when there is a timeout receiving a response [PMTUd] if PMTUd is rescheduled, then enforce a new run the usleep between each PMTUd run will give time to the other threads to grab the wrlock [PMTUd] make PMTUd force_run notification more quiet [PTMUd] if any threads receives a EMSGSIZE outside of a PMTUD run, force PTMUd to run Scenario: node X has MTU Y on the interface and application is sending packets with size >= Y. The interface MTU is suddenly reduced to < Y Before this change, the kernel would be dropping packets till the next PMTUd run. After this change, the PMTUd will be informed that it has to rerun (overriding the pmtud_interval), reducing the packet drop to a minimum. How to test: force knet_bench to send 1500 size packets with ping_data (requires code change) and start it. reduce MTU on the interface from 1500 to 1300 (for example) Notice an immediate trigger of PMTUd run in debug mode Note: going up, from 1300 to 1500 will wait for the next PMTUd re-run as there is no immediate way to detect this change unless we start listening to kernel netlink sockets with libnl3 (investigation in progress but not critical enough atm). [PMTUd] rename pmtud_running to pmtud_waiting [PTMUd] by using a safe PMTU to init a link, we can shave down corosync membership creation by 25% tested on a 9 node cluster with 2 links each node, going down from 4 seconds to 3 seconds [PMTUd] Use kernel MTU information to determine next packet size during discovery Using this information we can, for good links (*), determine and verify the link MTU with 2 packets. * good links means: node X has MTU Y configured on a given interface. Any network object between node X and destination is capable of handling MTU >= Y. In no case the kernel will allow us to send packets > Y. [stats] report correct MTU for loopback links Merge pull request #108 from kronosnet/minor-fixes Minor fixes [tests] add missing break (detected by newer gcc) [tests] knet_bench: allow to specify protocol (UDP/SCTP) per link [tests] knet_bench: add machine parsable output this is done by adding prefixes to different messages to make it easier to grep/awk and changing format of [perf] to reflect cpghum syntax. [tests] knet_bench allow to configure PMTUd interval from CLI [tests] fix regression in knet_bench introduced accidentaly by 2052faab4376107fe99e8a3f955bbcbfa426f648 [openssl] drop unnecessary reference counter 2018-01-03 wferi Merge pull request #109 from kronosnet/lzo2 build: work around broken pkg-config file in lzo2 version 2.10 2018-01-03 Ferenc Wágner build: work around broken pkg-config file in lzo2 version 2.10 In principle this isn't specific to BSD, though 2.10 is only packaged for BSDs at the moment. 2018-01-02 Fabio M. Di Nitto Merge pull request #106 from kronosnet/openssl-fixes [openssl] fix openssl1.0 crash by implementing documented locking cal… [openssl] fix openssl1.0 crash by implementing documented locking callbacks openssl < 1.1.0 do not support natively multithread applications and need some extra help from the application itself to provide locking callbacks. https://www.openssl.org/docs/man1.0.2/crypto/threads.html https://www.openssl.org/blog/blog/2017/02/21/threads/ to test this fix is necessary to use knet_bench or corosync with openssl and lots of heavy load (perf benchmark) workload. Sooner or later the application will crash with some random tracebacks. after this patch, the crash cannot be reproduced anymore. tested using 9 nodes x2 active/active links all running corosync + cpghum 2017-12-19 Fabio M. Di Nitto Merge pull request #99 from kronosnet/skip Skip tests if socket buffers are inadequate 2017-12-19 Ferenc Wágner Remove spaces after function names Anything that stops operation and returns an error should be log_err Document knet-specific errno values More fine-grained error handling tests: skip if socket buffers are too small tests: replace open coded error handling with knet_handle_start() calls Done by: $ sed -i -e '1h;2,$H;$!d;g' -e 's/knet_h = knet_handle_new(1, logfds\[1\], \(KNET_LOG_[A-Z]*\));\n\n\tif (!knet_h) {\n\t\tprintf("knet_handle_new failed: %s\\n", strerror(errno));\n\t\tflush_logs(logfds\[0\], stdout);\n\t\tclose_logpipes(logfds);\n\t\texit(FAIL);\n\t}/knet_h = knet_handle_start(logfds, \1);/g' libknet/tests/*.c tests: introduce knet_handle_start helper Report ENAMETOOLONG if socket buffers are too small 2017-12-19 Fabio M. Di Nitto Merge pull request #103 from kronosnet/mtu-lock [PMTUd] fix external API and PMTUd interaction 2017-12-18 Fabio M. Di Nitto [PMTUd] fix external API and PMTUd interaction The problem: PMTUd can take a long time to release the global read lock, mostly due to the pthread_cond_timedwait required to ack/nack packets from the other hosts. This delay could block any wrlock operation for several seconds if not more. The solution: each call to the global pthread_rwlock_wrlock has been changed to a wrapper that will notify the PMTUd to interrupt its operations (and restart) first, then get a global write lock that is queued as soon as PMTUd is going out. This solution also improves a lot shutdown speed. How to test: This is not super simple to test and verify. I used 2 VMs with known MTU of 1500. Start knet_bench on both (normal ping_data -C is more than enough). Once they have established data exchange, change the MTU on one of the nodes to 1600 (or higher). This should guarantee that the PMTUd process will take a very long time to complete. First verify that the PMTUd process takes several seconds. Once the next PMTUd run starts, hit ctrl+c on the node that is executing the PMTUd and the process should exit much faster than before this patch. Merge pull request #104 from kronosnet/spelling Spelling fixes 2017-12-18 Ferenc Wágner Fix typo and capitalization Fix grammar (allow requires an object) 2017-12-15 Fabio M. Di Nitto Merge pull request #102 from kronosnet/clang Clang [build] define conditionals only once [build] workaround BSD / clang breakage [build] fix clang builds Merge pull request #101 from kronosnet/mtu2 [PMTUd] fix multiple issues and stability problems 2017-12-14 Fabio M. Di Nitto [PMTUd] fix multiple issues and stability problems - resolve locking issue with thread_heartbeat that was causing spurious up/down link event. In the event of a PMTUd run taking too long, the heartbeat thread could hang for much longer than ping_timeout. Use backoff_mutex to sync between threads instead of the global lock. - pause the DATA tx thread when sending any PMTUd related packets. Similar method as knet_send_sync, using the tx_mutex, allows a much more stable communication between nodes without any visible performance hit. - calculate higher timeouts when using crypto to improve stability - fix an odd race condition with the kernel where, during a single PMTUd run, the same packet size was marked both BAD and GOOD (via EMSGSIZE) by the kernel. That situation would cause our PMTUd to run away and calculate bad values. - add a minor usleep between sending PMTUd packets to give time to the kernel to make its own mind about the link PMTU. This is based on average latency. - since PMTUd can take several seconds before completion, use the "end time" to record the last run vs the start time. - fix a major issue in sending PMTUd reply where an errno was not being passed down the link layer and would cause the RX thread to block forever. 2017-12-09 Fabio M. Di Nitto Merge pull request #98 from liu4480/add-debuginfo allow to choose whether to build debuginfo packages or not 2017-12-08 Bin Liu allow to choose whether to build debuginfo packages or not 1. If not "--enable-rpm-debuginfo" or "--disable-rpm-debuginfo" is specified, follow the system default behavior 2. If set, then build debuginfo package or not build based on the flag. 2017-12-05 Fabio M. Di Nitto Merge pull request #96 from kronosnet/modules2 Modules2 2017-12-04 Ferenc Wágner build: remove useless assignment build: use AS_HELP_STRING for consistent formatting build: recover original option ordering build: move more repetitive module work into our Autoconf macro Check ABI version on module load 2017-12-04 Fabio M. Di Nitto [tests] drop obsoleted tests that are broken after moving to the new modules format [tests] make module tests dynamic by using internal knowledge of what is built or not drop all related #ifdef as well 2017-11-30 Fabio M. Di Nitto [modules] update valgrind BSD exceptions [modules] spec file: fix directory shipping [modules] update spec file to ship modules in separate rpms + meta packages 2017-11-30 Ferenc Wágner Unify compress and crypto module loaders Unify crypto module initialization functions Taken from Fabio M. Di Nitto. libknet.h uses ssize_t, include its definition directly This fixes the BSD build. Pass log_msg to the plugins Switch over all plugins to the module system 2017-11-29 Ferenc Wágner Make the bzip2 compress plugin a proper module Our current practice of dlopening foreign shared libraries is problematic for several reasons: * not portable: modules and shared libraries can be different object types * dependency information is invisible (our canaries mostly solve this) * hardwiring SONAMES breaks on transitions (KNET_PKG_SONAME solves this) * symbol versioning information is lost (theoretically solvable) The preferred way out is generating dynamically loaded private modules from the main source, which then rely on the dynamic linker to load the external symbols as usual. Separate compress_model_t definition into a new header file We'll use this type together with the zlib headers, which clash with our compress.h because both define a function named compress. 2017-11-28 Fabio M. Di Nitto Merge pull request #94 from kronosnet/sock [transport] make the new derooted sock management less chatty [transports] add logging around setsockopts that have specific per distro/include requirements this is helpful to determine if a given option has been built-in and/or activated properly [transport] add missing include files and really fix 86c81a32cfccde1dea [transport] consistency is not optional :-) [transport] fix errno handling [transport] make the new derooted sock management less chatty 2017-11-27 Fabio M. Di Nitto Merge pull request #91 from kronosnet/no-root Enable testing without root privileges 2017-11-27 Ferenc Wágner Detailed error reporting about socket buffer sizes 2017-11-26 Ferenc Wágner Remove the need_root() function The point is not having root privileges, but having high enough socket buffer caps. Very few of the test can run without it. 2017-11-25 Ferenc Wágner Don't require root privileges unless necessary On Linux, if /proc/sys/net/core/[rw]mem_max are set to at least 8388608 (KNET_RING_RCVBUFF), setting the socket buffer sizes doesn't require root privileges. FreeBSD uses the kern.ipc.maxsockbuf sysctl MIB variable for capping user buffer requests. Linux doubles the requested amount for administrative overhead, but FreeBSD does not, so we can't be too strict when checking the results. 2017-11-25 Fabio M. Di Nitto Merge pull request #93 from kronosnet/dynamic-pong Dynamic pong [tests] workaround buggy valgrind-freebsd implementation. valgrind is not officially supported on FreeBSD. The current valgrind-freebsd port is maintained separately and it's lagging several releases behind vs upstream valgrind. When running knet+openssl+valgrind on x86_64, with high pthread_cond_timedwait configuration, valgrind appears to be stuck tracing openssl internal operations to the point that the internal knet RX thread is not scheduled for minutes. At that point all processes will go kaboom (heartbeat, PMTUd, etc) almost constantly failing this test. Given that all other architectures and OSes can run this test happily, I am adding this exception to the test suite, to be re-evaluated in future if newer versions of valgrind will be available on FreeBSD. 2017-11-24 Fabio M. Di Nitto [tests] don't duplicate test [tests] use a new tunable functions to wait for packets [tests] when running under valgrind, allow a lot more time for host to be reachable [links] fix description [links] add pong_timeout backoff mechanism to deal with high latency links Merge pull request #92 from kronosnet/timer-fixes Timer fixes [links] extend knet_link_set_ping_timers ABI to be ready for future expansion [tests] implement common and more flexible wait_for_host implementation [docs] update header to be more specific about ping timers and relative man pages [PTMUd] don't harcode 2 seconds timeout to receive a PMTUd response from the remote node 2017-11-23 Fabio M. Di Nitto Merge pull request #87 from chrissie-c/balance_crypt_stats stats: Make the crypt_packet stats balance 2017-11-23 Christine Caulfield stats: Only collect crypt over stats for received data packets This is more for consistency really. The TX code only accumulates the crypto overhead for data packets because it's not worth adding all the accounting code around the ping/pong & pmtu transmitters. stats: Make the crypt_packet stats balance The tx_crypt stats only included data packets whereas the rx_crypt stats included everything, because the TX was done in different places and the RX in just the one. I've added a 'stats_extra' structure so that the other threads can update their own stats without extra locking, the get_stats call adds them in as necessary. 2017-11-21 Fabio M. Di Nitto Merge pull request #90 from kronosnet/soname build: determine the plugin SONAMEs automatically 2017-11-20 Ferenc Wágner build: determine the plugin SONAMEs automatically Most importantly, this avoids dlopening libcrypto.so, which is a symlink in the OpenSSL development packages on Linux. Rather, we use the SONAME of the first library added by pkg-config, which seems to work well across the board. The strange case is NSS, which ends up using libssl3.so on CentOS, Fedora and RedHat, but libnss3.so on Debian, FreeBSD and Ubuntu. The tests pass regardless, so this might be tolerable. 2017-11-17 Fabio M. Di Nitto Merge pull request #89 from chrissie-c/knet_bench_stats [knet_bench] Add option to display stats 2017-11-17 Christine Caulfield [knet_bench] Add option to display stats 2017-11-15 Fabio M. Di Nitto Merge pull request #88 from kronosnet/nss-workaround [build] workaround memory alignment bug in nss in combination with va… [build] workaround memory alignment bug in nss in combination with valgrind on Linux/i386 Over the past few weeks, we noticed CI failing on i386 when running make check-memcheck (test suite executed with valgring). Those failures were not consistent and sometime random. After some heavy debugging it turns out that those failures are a combination of libnss3 internal memory allocator bug and valgrind internal memory allocator. Current libnss3 memory allocator expects a 16 bytes memory aligned buffer and in the event the memory is not aligned, it would round it. The issue is that, in performing this rounding, nss does not track properly the original address of the memory, and the equivalent of subsequent free of this buffer would fail, leaking memory and potentially accessing memory not allocated or corrpting memory. The nss bug is already fixed upstream by commit: changeset: 13557:52e38f913220 Valgrind on Linux/i386 memory allocator can return memory that is not aligned to the 16 bytes, contrary to what malloc/glibc does. Valgrind was returning (somehow randomly) a non aligned buffer to nss triggering the nss bug and the leak. By forcing valgrind to use aligned memory, nss bug does not trigger and there is no memory leak during the execution of the test suite. The memory leak never triggered when running knet in normal conditions. 2017-11-14 Fabio M. Di Nitto Merge pull request #86 from kronosnet/pthread build: adopt ax_pthread.m4 to provide all necessary flags 2017-11-14 Ferenc Wágner build: adopt ax_pthread.m4 to provide all necessary flags Previously we only linked with -lpthread, but for example the -pthread GCC option provides a CPP define as well. 2017-11-14 Fabio M. Di Nitto Merge pull request #85 from wferi/libcrypto We only use the crypto part of OpenSSL 2017-11-14 Ferenc Wágner Revert "build: link as needed" This reverts commit e3c019fde32f78bf2e3a9bcf6c2fff93245708a0. Once LIBS is kept empty and OpenSSL isn't linked to in full, overlinking shouldn't be a problem anymore. build: only kronosnetd needs PAM, so keep the linker options separate build: remove unused function checks These have no corresponding HAVE_X checks in the code. build: the check_lib_no_libs function was unused The optional arguments of AC_CHECK_LIB work good enough. build: don't add the dl library to LIBS, keep it in dl_LIBS only This leaves us with an empty LIBS in the most common cases. 2017-11-13 Ferenc Wágner build: clock_gettime does not require the rt library for glibc 2.17+ So stop overlinking with -lrt on modern systems. build: we only use the crypto part of OpenSSL 2017-11-13 Fabio M. Di Nitto Merge pull request #84 from kronosnet/api-fixes Api fixes [libknet] fix comments and update man pages [libknet] add padding to get *_info structs [tests] fix tests differently [tests] fix build on i386 [tests] add api_knet_send_crypto [transports] knet_get_transport_list should behaves as knet_get_compress/crypto_list [tests] fix error message in api_knet_get_compress_list.c [docs] update copyright and manpagea [compress] move knet_handle_get_compress_list to knet_get_compress_list now matches the same API as get_transport_list [crypto] move knet_handle_get_crypto_list to knet_get_crypto_list now matches the same API as get_transport_list [logging] don't hardcode the max, use max value for the typedef [transports] make KNET_MAX_TRANSPORTS = UINT8_MAX now that the APIs allow holes in the IDs this way we don't need to break ABI if we add new transports later on. [transports] cleanup API calls knet_get_transport_id_by_name, knet_get_transport_list and knet_get_transport_name_by_id don't need a handle to be functional as they only access build-time info that don't change at runtime. [transports] move to use the same internal API as crypto and compress modules Merge pull request #83 from wferi/canary Automatic canaries 2017-11-13 Ferenc Wágner Extend the preprocessor schema to the NSS crypto plugin Extend the preprocessor schema to the OpenSSL crypto plugin 2017-11-12 Ferenc Wágner Validate dynamic symbol prototypes Practically, all imported symbols are functions returning an int, but the LZMA API defines it as an enum. Extend the preprocessor schema to all compression plugins Use the preprocessor for the zlib symbol remapping as well This ensures that the canary uses all dynamically loaded symbols, enabling precise versioning of the optional dependencies. Use the preprocessor to reference all remapped zlib symbols in the canary 2017-11-11 Ferenc Wágner Switch libknet/compress_zlib.c to using the remap_symbol helper Define a common helper function for symbol remapping git ignore libknet/{compress,crypto}_canary 2017-11-10 Fabio M. Di Nitto Merge pull request #80 from kronosnet/clang-fixes Clang fixes [clang] fix build on freebsd [tests] add exception for lzma on Ubuntu 17.10 i386 (clang) - take 2 [tests] add exception for lzma on Ubuntu 17.10 i386 (clang) Merge pull request #81 from wferi/canary Generate dependency information for plugin libraries 2017-11-10 Ferenc Wágner Generate dependency information for plugin libraries 2017-11-09 Fabio M. Di Nitto Merge pull request #79 from wferi/git-version-gen build: update git-version-gen 2017-11-09 Ferenc Wágner build: update git-version-gen This newer version copied from libqb does not speak -dirty if the tarball is extracted into a GIT repository, which happens during packaging. https://lists.gnu.org/archive/html/coreutils/2011-02/msg00069.html 2017-11-07 Fabio M. Di Nitto Merge pull request #78 from kronosnet/docs Docs [docs] update man pages with proper copyright info 2017-11-06 Fabio M. Di Nitto Add copyright to the README file [docs] update README file [docs] update man pages with copyright info [docs] drop TODO list [docs] remove empty publican docbooks 2017-11-06 Christine Caulfield doc: Add copyright section to man pages 2017-11-06 Fabio M. Di Nitto [docs] add copyright to more files Cleanup export sym naming 14:42 < chrissie> Here's the quote in the manual if you need it https://ftp.gnu.org/old-gnu/Manuals/ld-2.9.1/html_mono/ld.html#SEC25 14:42 < fabbione> lol, i trust you :-) 14:51 < chrissie> most I wondered if you wnated a reference for the commit [docs] more copyright fixing Merge pull request #76 from kronosnet/docs [docs] Update copyright info and add tools to automate it [docs] fix git startdate to follow file renames and set enddate to this year [docs] Update copyright info and add tools to automate it Merge pull request #75 from kronosnet/docs [docs] update COPYRIGHT [docs] update COPYRIGHT 2017-11-02 Fabio M. Di Nitto Merge pull request #71 from chrissie-c/parallelise_tests tests: Parallelize tests 2017-11-02 Christine Caulfield tests: Fix [s]printf of uint32 tests: Parallelize tests Randomize (using PID) the port numbers used for the tests so they don't clash. 2017-11-02 Fabio M. Di Nitto Merge pull request #74 from chrissie-c/mark-link-configured-later link: Only mark a link as 'configured' when it actually is 2017-11-02 Christine Caulfield link: Only mark a link as 'configured' when it actually is If the transport-specific config fails then the link can be left unconfigured, but marked as 'configured'. I've only moved the setting of 'configured' flag itself rather than the rest of the values in case the transport-specific code wants to change them for any reason. 2017-11-01 Fabio M. Di Nitto Merge pull request #73 from kronosnet/build-fixes Build fixes [build] fix make distcheck on bsd machines make sure to clean temporary files created by sed on the way of building [build] fix git dirty detection [build] fix changelog generation [build] fix make distcheck target [build] don't ship prebuilt spec file with release tarball [build] disable kronosnetd build with distcheck Merge pull request #72 from kronosnet/build-fixes [rpmbuild] fix BuildRequires to allow rpmbuild for both Fedora and Op… [rpmbuild] fix BuildRequires to allow rpmbuild for both Fedora and OpenSUSE based distros 2017-10-31 Fabio M. Di Nitto Merge pull request #70 from wferi/dist Add the notes and the docbook to the distribution 2017-10-31 Ferenc Wágner Add the notes and the docbook to the distribution 2017-10-31 Fabio M. Di Nitto Merge pull request #69 from kronosnet/build-fixes [bsd] fix builds and testsuite on BSD x86_64/i386 [bsd] fix builds and testsuite on BSD x86_64/i386 I don't particularly enjoy the use of host detection in configure.ac, but there is no easier way to set the library names for dlopen without some complex and fragile scripts. BSD valgrind + nss complains about some non-initialized memory access. being internal to nss we can ignore it. on BSD i386 size of time_t is int32_t vs all other platforms where is int64_t. BSD doesn't appear to be offerign any PRIx conversion and since it's contained within the testsuite only (for printf purposes), cast and get away with it. add check to avoid building libtap on BSD for now. 2017-10-30 Fabio M. Di Nitto Merge pull request #68 from chrissie-c/dont-send-channel-32 [tx] Don't send to channel 32 if there are no channels available 2017-10-30 Christine Caulfield [tx] Don't send to channel 32 if there are no channels available For some reason this only triggers on FreeBSD (that we've seen). 2017-10-30 Fabio M. Di Nitto Merge pull request #67 from kronosnet/minor-cleanup [libknet] rename transport_info to knet_transport_info [libknet] rename transport_info to knet_transport_info match the namespace of all our struct. This is an API change, but doesn't affect ABI. Merge pull request #66 from kronosnet/doxyfication Doxyfication [man] update man pages white space cleanup 2017-10-30 Christine Caulfield doc: Fix minor oddities in the doxygen comments of libknet.h host_status_change_notify_fn() still generates a duff man page but that seems to be a bug in doxy2man 2017-10-30 Fabio M. Di Nitto add autogenerated man pages and build target to update them on demand 2017-10-28 Fabio M. Di Nitto [docs] generate libknet .3 man pages at build time 2017-10-27 Christine Caulfield doc: First attempt at Doxyfying libknet.h 2017-10-27 Fabio M. Di Nitto Merge pull request #65 from kronosnet/release-fixes Release fixes [release] fix scp call for publishing files [release] update .gitignore [release] sign all tarballs and file list Merge pull request #64 from kronosnet/libtap-fixes [libtap] reduce size of max updownpath to adjust for max action, / an… 2017-10-26 Fabio M. Di Nitto [libtap] reduce size of max updownpath to adjust for max action, / and max tapdev name size Merge pull request #63 from kronosnet/nodebian [build] drop debian support from upstream [build] drop debian support from upstream On Debian HA team request, we are dropping the debian/ dir Merge pull request #61 from kronosnet/bsd [build] update Makefile.am to work with BSD make [build] update Makefile.am to work with BSD make Merge pull request #62 from chrissie-c/new-apis2 libknet: Add APIs to query crypto & compression models 2017-10-26 Christine Caulfield libknet: Add APIs to query crypto & compression models Also add API call to clear the handle and link stats 2017-10-26 Fabio M. Di Nitto Merge pull request #60 from wferi/autotools Autotools 2017-10-26 Ferenc Wágner configure: initialize pkg-config with its standard macro configure: remove some superfluous checks We've got no C++ code in here. No need for ranlib when using libtool. 2017-10-25 Fabio M. Di Nitto Merge pull request #59 from kronosnet/ci-fixes Ci fixes [build] enable valgrind automatic supressions generation during make check-* this is useful to create exception when false errors are found on CI run [tests] add exception for lzma on Ubuntu 17.10 i386 Merge pull request #58 from liu4480/run-autogen add "runautogen" option for compiling 2017-10-25 Bin Liu add "runautogen" option for compiling 2017-10-19 Fabio M. Di Nitto Merge pull request #56 from kronosnet/fabbione-cleanup [testing] drop travis support as we are moving all builders to https:… [testing] drop travis support as we are moving all builders to https://ci.kronosnet.org/ Merge pull request #57 from kronosnet/fabbione-udp-fixes [udp] fix incorrect return code [udp] fix incorrect return code UDP TX is tricky and racy and in theory TX never fails, as UDP is allowed to drop packets internally anyway. In some race condition situations, we can endup in a situation where: - thread X attempts to send a packet - socket receives an ICMP back (for whatever reasons) generated by some other random thread/packet combo - thread X detects an error in TX and decode the error (correct) pre patch: - thread X would treat that as error and drop the packet post patch: - thread X should simply resend the packet simply also the ordering of parsing socket error code to make it easier to read 2017-10-18 Fabio M. Di Nitto Merge pull request #53 from digimer/master Added 'gcc' to .spec BuildRequires 2017-10-18 Digimer * Added a requirement for 'gcc' as that is not installed on fedora minimal by default, breaking rpmbuild's build attempt. 2017-10-16 Fabio M. Di Nitto [global] use memmove instead of memcpy 2017-10-15 Fabio M. Di Nitto [build] more casting fixes [build] more signed/unsigned fixes [build] fix some signed/unsigned declarations [build] allow make check-memcheck and check-hellgrind only if valgrind is avalable [build] bump minimum requirements for automake this is required for distclean + subdir-object to work properly 2017-09-26 Fabio M. Di Nitto [tests] fix build and cast aligment on Debian/sparc64 unstable (gcc7) 2017-09-25 Fabio M. Di Nitto [build] cleanup whitespace [build] disable poc code by default 2017-09-21 Fabio M. Di Nitto [dlopen] drop support for unload of libraries [openssl] check for errors when initializing openssl1.1 [build] fix build on freebsd11 (and others) 2017-09-19 Fabio M. Di Nitto [tests] fix build with no nss [tests] move dump_all to callback section [tests] add exception for lzma on Debian Unstable [test] drop useless library count check [crypto] add support for openssl1.0.2 and openssl1.1.0 to accomodate Debian unstable Please note that openssl1.1.0 needs more testing 2017-09-19 Christine Caulfield [host] Never mark loopback-connected hosts as unreachable There's no point in checking the links of a host with a LOOPBACK link - it's always up and reachable. 2017-09-19 Fabio M. Di Nitto [openssl] update libknet.h to document openssl support [openssl] extend lib load/unload to cover for openssl [openssl] filter "none" from *getbyname openssl functions [openssl] drop unrequired OPENSSL_config [openssl] revert debugging code [openssl] go back to the original minimal keysize [openssl] move to dlopen model [nss] protect shutdown from segfaults in case we cannot resolve all syms [openssl] add api_knet_handle_crypto test [openssl] drop incorrect keylen management code [crypto] add openssl support (first cut) still needs dlopen and memcheck love + sync with nss key size [cryto] nss: add namespace to internal functions 2017-09-18 Fabio M. Di Nitto [build] fix casting with gcc-7 (debian unstable) [debian] fix tests/ linking with debian unstable [debian] fix build deps (pass 3 or 4) [debian] be more specific about builddeps [debian] fix build dependency in control file 2017-09-14 Fabio M. Di Nitto [test] fix conditional build of int_lib_load_unload [global] bump lib soname to 1.0.0 [dlopen] compress: drop unnecessary tracker and use init array instead [compress] move tracking update into a write lock context [dlopen] compress: fix load/unload of libraries and add extensive test each handle should only unload libraries that has loaded or referenced on exit, attempt to close all libraries from that handle on compress_fini close only the one that's open [compress] make sure to unload resources before re-init the new config [global] rename link.(c|h) to links.(c|h) as they conflict with global /usr/include/link.h [compress] split check_lib_init into 2 functions [crypto] simplify lib init code and fix error handling also update the valgrind exception file [dlopen] move all dlopen lib names at the top of the respective files makes it easier to figure out library supported version [dlopen] add suppress entries for internal NSPR/NSS memory leaks (part 2) [docs] fix a typo [dlopen] fix crypto/compress internal name space for similar functions [dlopen] add suppress entries for internal NSPR/NSS memory leaks [dlopen] use RTLD_NOW instead of RTLD_LAZY to avoid perf hit on fast path [crypto] partially revert and partially fix b22ef0ca65d687dbc nss/nspr still leaks approx 2kb on unload, but it's much cleaner now than the many Kb before those fixes. https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/Memory_allocation [handle] remove knet handle tracker reload of shlib should be done on a per handle base to avoid complex global functions. For example, if you application is using 2 knet_h and you need to reload libbz2, then switch both knet_h to use another compression method (or none) and switch back to libbz2. Similar can be done with nss/openssl once openssl support is added. A global function to reload all library would a very complex error reporting mechanism and could leave some handles improperly configured. The application would have a better understanding of the needs at any given time. [crypto] fix nss unload/reload memory leak [dlopen] if the library is a symlink, display the symlink info to the final binary it cannot prevent symlink attacks but definetely helps to identify them [common] don't hardcode max path size [dlclose] add logging messages around unloading libraries [common] use a standard fn to dlopen libraries and print load path info might be useful to detect LD_PRELOAD attack [crypto] nss: be more clear about shut down/unload situation [tests] fix crypto test to deal with new nss dlopen space [nss] clear error before dlopen [nss] warn the users that library cannot be unloaded [crypto] fix libnss horror show to avoid leaking memory libnss simply can't be unloaded at runtime as it leaks memory. switch back to the hold atexit code and don't allow unloading of the library. [crypto] handle properly errors reported from .init functions [crypto] fix libnss unload crash and remove that atexit horror show [crypto] fix libref count the same way as compress [compress] centralize handling of libref in compress abstraction layer the complexity come from a mix and match of those modules that have/don't have .is_init/.init functions, and makes handling more delicate. This commit ensures that: - library is loaded only once (makes load_lib mandatory for every modules) - use knet_h->compress_int_data to track per handle init. modules providing .is_init,.init don't need set anything there unless they need to. - modules don't need to track libref anymore but they must provide a unload_lib/load_lib. - library are closed only once libref is 0. [crypto] move libnss to a dlopen model (pass 1) unload and exit will coredump for now [dlopen] implement dlopen rate limit to avoid RX DoS [global] update notes to package maintainers [compress] fix internal ref count for library unload [compress] switch to new load_lib / unload_lib API [compress] fix error message to be more explicit [dlopen] start to cleanup the compress module API [build] consistent error message [build] move hardcoding of linking libraries in one place and enforce better check [compress] move bzip2 to dlopen model [compress] cleanup dlerror before dlopen for lzma [compress] cosmetic cleanup... (part 2) [compress] cosmetic cleanup [compress] move lzma to dlopen model [compress] move lz4 to dlopen model [compress] make sure lib handles are clean after dlclose [compress] move zlib to dlopen model [compress] fix library init to work for multiple knet_handle [compression] simplify lzo2 dlopen code handling code is still broken when using multiple knet_handle [compression] move lzo2 to dlopen model and fix internal api this commit breaks all other compression modules for now [test] add the test file [compress] only initialize compress library on demand [test] don't run api_knet_handle_new_limit by default it takes more than 20GB of ram and special ulimit settings. [handle] implement open knet_handle tracker limit the number of open knet_handle to UINT8_MAX for now. UINT8_MAX requires approx 20GB of RAM only for init and more than 1024 fds (tune your ulimit as necessary) without taking into account fds required for links 2017-08-24 Christine Caulfield libknet: some FreeBSD compatibility fixes. Also add support for KNET_LINK_FLAG_TRAFFICHIPRIO on FreeBSD 2017-08-21 Chrissie Caulfield add knet_handle_get_stats (#49) * stats: Add compression and crypto statistics New stats structures shows how effective the compression is, and also the compression/decompression overhead time. Crypto stats show the byte overhead of the crypto library as well as the time taken to sign/encrypt and decrypt the packets. Signed-off-by: Christine Caulfield 2017-08-21 Fabio M. Di Nitto [build] drop -lpthread from pkg-config 2017-08-18 Fabio M. Di Nitto [tests] fix knet_bench usage [build] fix several hanging bits after compress/crypto optional builds [build] allow optional build of nss crypto support [build] rename nsscrypto to crypto_nss and match compress module names [build] fix SCTP support build 2017-08-17 Fabio M. Di Nitto [compress] fix build to allow --disable-compress-all --enable-compress-$foo disable all but build $foo... [compression] change test from ERROR to SKIP [compress] report error from knet_handle_compress api test if there is no zlib builtin [compress] update api_knet_send_compress test [compress] split init and config functions 2017-08-17 Christine Caulfield [compress] fix some Itaglish & a trailing space 2017-08-17 Fabio M. Di Nitto [compress] fix test suite build [compress] allow conditional builds of compress modules [compress] prepare internal structs and code to allow selection of compression models [compress] add pkg-config support for lzo2 recent versions of lzo2 have pkg-config support. Use it when available or fall back to old detection method. [compress] add bzip2 support [compression] update libknet.h for lzma [compress] add lzma support update also the memcheck exception file for valgrind. those updates are strictly for lzma internal calls and they work on both rhel7 and fedora 25. [compression] remove debugging code [compress] add support for lzo2 compression [testing] add knet_send with all crypto modules test [compress] add support for threshold (don't compress packets smaller than X) [compress zlib] fix log levels around [compress lz4] fix log levels around [compress] remove stray debugging entry [compress] add support for lz4 and lz4hc compress mechanism [compress] fix header define [packaging] add explicit build dependencies [compress] add userdata compression support - implement generic plugin for compression - add zlib compression support 2017-08-07 Christine Caulfield loopback: Optimise loopback sending If a packet is only sent via loopback then don't bother fragmenting or encypting it. 2017-08-04 Fabio M. Di Nitto Merge pull request #48 from chrissie-c/loopback-single loopback: Only allow 1 link to localhost if loopback is used. 2017-08-04 Christine Caulfield loopback: Fix some errno assignments that should have been to savederrno 2017-08-03 Christine Caulfield loopback: Only allow 1 link to localhost if loopback is used. If loopback is configured as a link transport to localhost then no other transport makes sense and could actually cause duplicate packets to be received. So now it's enforced and tested. 2017-08-03 Fabio M. Di Nitto [build] minor "make my OCD more quiet" cleanups [tx] use loopback logging [crypto] use unique name to avoid collision with other plugins [logging] add loopback nice name 2017-08-02 Fabio M. Di Nitto [transport] minor nitpick 2017-08-01 Christine Caulfield libknet: Add LOOPBACK transport KNET_TRANSPORT_LOOPBACK is a transport designed for high-speed communication internal to the local node. Data packets are copied directly from the sendfd to the receiving fd. The calling program must make sure that data is removed from the receiving fd in a timely manner. Reviewed-by: Fabio M. Di Nitto 2017-07-31 Fabio M. Di Nitto Merge pull request #47 from jnpkrn/build+init Build+init cleanups 2017-07-31 Jan Pokorný [init] service file: drop no longer available ControlGroup= option Support for that was dropped in systemd 205[1]. To be able to get RT priority regardless of the runtime control groups (Linux-specific), there's a precedent solution for corosync[2], should it be needed. [1] https://lists.freedesktop.org/archives/systemd-devel/2013-July/011679.html [2] https://github.com/corosync/corosync/pull/231 [build] configure: avoid confusion wrt. what's enabled by default 2017-07-30 Fabio M. Di Nitto [build] use newly create kronosnet release team gpgkey [build] enable release script / publishing bits [build] make maintainer-clean clean also after a release [build] add Cluster Release Team signing key 2017-07-28 Fabio M. Di Nitto Merge pull request #46 from jnpkrn/build-release.mk [build] release.mk: adapt recent changes from libqb (fence-agents) 2017-07-28 Jan Pokorný [build] release.mk: reflect current release publishing practice Core author(s) had previous affiliation with project hosting on fedorahosted.org, but since the sunset of that hosting[1], it's no longer an option. Unfortunately, publish-by-scp convenience of fedorahosted.org won't fly with GitHub that insists on the (primarily REST) API to get things automated, so update "publish" recipe to that effect, providing some hints on possible adaption of the process. Also, turn "git push --tags" to a bit safer "git push --follow-tags" for good measure. [1] https://lists.fedoraproject.org/archives/list/announce@lists.fedoraproject.org/message/BWSMCGZPPNG3JOCFQ6Z74MIBU7FG3KGB/ [build] release.mk: simplify the default goal, declare .PHONY targets The default goal line now serves mainly the documentation purpose, otherwise it could be squashed down to "sign" target. [build] release.mk: move soft guard for no GPG key up the supply chain It also means the missing key specification is now a hard error when ASCII armored files are to be created (directly or by other rules). Leveraging the previous commit, it is now unnecessary to guard against release mode not being requested, so this guard is dropped completely in the transition, allowing for the full-fledged (incl. GPG signing) release delivery testing without mangling with in-tree tags. To allow "sign" target to (possibly eventually) become the implicit goal even if "gpgsignkey" variable is explicitly unspecified, make it depend on $(deliverables). [build] release.mk: fix no-release conflict (implied-required version) Previously, generating custom versioned tarballs without requiring a "release" (which comprises also tagging hence in turn making the implied version match the required one as desired) would not allow to test the release routines thoroughly incl. GPG signing. Unfortunately, allowing for a temporary tag is prone to accidentally slipping it to the distributed tree, so that should rather be avoided Now it can. Another step to blur the boundaries between whether "release" is specified or not is to make checksums file generated unconditionally. Also both pre-existing and the newly created (per the above scenario) "overhead" files arising from make/release.mk are git-ignored properly. [build] release.mk: simplify/generalize GPG signing rule ...especially in case we want to easily switch to signing also particular archives, which may be a reasonable expectation from some downstream/distro maintainers (as demonstrated with the libqb's example[1] where this changeset was picked from a followup PR[2], respectively its adaptation for fence-agents[3]). Since this commit, such a change is a matter of only swapping a prerequisities specification for "sign" target per the commented out suggestion. For that to work reliably even after such a change, for the complete dependency backtrack, though, a pre-existing issue with parallelized work on the prerequisites needs to be fixed first. The (currently hypothetical) problem is "tarballs" (phony) target missing in the linearized supply chain for the signed tarballs, as opposed to supply chain for the checksums file that can be worked on in an (unordered!) interleave with the work on these signed tarballs. Hence fix this (currently hypothetical) race condition by stretching "tarballs" target as progress barrier (using it as a prerequisite) uniformly for all the deliverables that are going to be signed (as opposed to the currently only "sha256" target). And when at it, make "deliverables" variable list them conveniently (along with reformatting definitions of other variables for uniformity). [1] https://github.com/ClusterLabs/libqb/issues/145#issuecomment-306640694 [2] https://github.com/ClusterLabs/libqb/pull/260 [3] https://github.com/ClusterLabs/fence-agents/pull/123 2017-07-24 Christine Caulfield libknet: Small fixes for errors found by GCC 7.1 2017-07-18 Christine Caulfield libknet.h: change some status values from bitfield to uint8_t Bitfields are a pain and you can't take the address of them which makes it hard for cosoync stats to deal with the values Reviewed-by: Fabio M. Di Nitto 2017-07-12 Fabio M. Di Nitto [nss] expand error reporting with proper error strings thanks Honza for adding some of them in the previous commits [crypto] set default case for key_type and avoid build warnings when key_type is unknown, gcc complains that cipher can be used uninitialized. add default case to catch wrong callers and avoid build warning [crypto] cleanup inconsistent whitespacing Merge pull request #44 from jfriesse/fips-enable Fips enable 2017-07-11 Jan Friesse [knet.h] Reduce KNET_MIN_KEY_LEN By reducing KNET_MIN_KEY_LEN we allow FIPS enabled system using NSS to import (unwrap) the key. [nsscrypto] Use different method to import key PK11_ImportSymKey doesn't work when FIPS is enabled because NSS is targeting to FIPS Level 2 where loading of unencrypted symmetric key is prohibited. FIPS Level 2 is hard to achieve without breaking compatibility so patch implements "workaround" to make NSS behave like FIPS Level 1 (where is allowed to load unencrypted symmetric key). Workaround is about using temporal key to encrypt corosync authkey in memory and then to unwrap it into valid NSS key. Also when FIPS is enabled, it's not currently (nss 3.30.2) possible to unwrap key larger than 256 bytes. [nsscrypto] Refactor symmetric key importing 2017-06-24 Fabio M. Di Nitto [stats] add doc details in the stat header [link] switch link_get_status to write lock previous to the introduction of detailed link stats, the values contained in the status structure would hardly change over time making a rdlock more than enough to get the right data. Now that the values are updated constantly, we need to remove the window of thread X writing values and thread Y to read them while being written. 2017-06-20 Chrissie Caulfield libknet: Add link stats (#43) * libknet: Add link stats This patch also includes an ABI change to knet_link_get_status() (as well as returning a load of stats) that should allow us to add more stats at a later stage without ballooning memory use with pads or crashing earlier-linked programs with structure overwrites. kronosnetd compiles but has not been tested... 2017-06-12 Fabio M. Di Nitto [rx/tx] Increase RX buffers to 512 (new-tx branch) latency is still much lower than current udpu and throughput for small packets is much better [kronosnetd] fix build with new link flags 2017-06-09 Christine Caulfield libknet: Add link flag for 'hi' (interactive) priority Rather than make it the default, this is now a link option. Corosync needs it but other applications probably will not. Reviewed-by: Fabio M. Di Nitto 2017-06-08 Christine Caulfield libknet: Add 'flags' parameter to knet_link_[gs]et_config() I have at least 1 flag planned but I'm sure there's plenty of scope for more. Also trying to get the API changes out of the way ready for the cluster summit in September. 2017-06-04 Fabio M. Di Nitto Reduce TX memory further down by some MB [tx/rx] reduce memory footprint and latency reduce memory usage by approx 16MB and drop min/avg latency for some workloads 2017-06-03 Fabio M. Di Nitto Resolves signed-unsigned comparison warnings Patch based on: commit 730a8b6a56820741b361e0907eecb6297a9f0348 Author: Michael Jones Resolves: https://github.com/fabbione/kronosnet/pull/40 2017-06-03 Michael Jones Fixes format=signedness related warnings Adds additional warning flags to the compiler, no new warnings unhidden 2017-05-26 Christine Caulfield [sctp] Don't hold global_rwlock while sleeping. The SCTP connect and accept threads can do small sleeps when there are errors. The global_rwlock should not be held while this is happening. While the lock is a read lock and doesn't block sending and receiving in other threads it can cause a deadlock on shutdown. [libknet] Unlock the global lock if logging pipe is full. If a write to the logging pipe fails (eg it's busy waiting on something else, I've seen this in corosync at shutdown) then we must unlock the global_rwlock or we'll never get out of this alive. Reviewed-by: Fabio M. Di Nitto 2017-05-11 Christine Caulfield [libknet] Add TC_PRIO_INTERACTIVE to sockets 2017-04-19 Fabio M. Di Nitto Merge pull request #36 from liu4480/remove-if-defined-i386 use PRIu64 to replace #if defined(__i386__) 2017-04-19 Bin Liu use PRIu64 to replace #if defined(__i386__) 2017-04-13 Christine Caulfield [libknet] Add some includes to allow FreeBSD compilation Reviewed-by: Fabio M. Di Nitto 2017-04-13 Fabio M. Di Nitto Merge pull request #35 from liu4480/type-fixes fix compile errors on i386/i686 for knet_bench.c Resolves: #34 2017-04-13 bliu fix compile errors on i386/i686 for knet_bench.c 2017-04-13 Fabio M. Di Nitto Merge pull request #33 from liu4480/type-fixes fix compile errors on i686 2017-04-13 bliu fix compile errors on i686 2017-03-17 Fabio M. Di Nitto [docs] update slids a tiny bit 2017-03-03 Fabio M. Di Nitto [tx] reduce memory usage and memmove on big pckts via better iovec mapping 2017-03-01 Fabio M. Di Nitto [build] fix clang errors this should fix https://github.com/fabbione/kronosnet/issues/32 2017-02-28 Fabio M. Di Nitto Merge pull request #30 from jfriesse/shadow-time [knet_bench]: Fix shadow declaration of time 2017-02-27 Jan Friesse [knet_bench]: Fix shadow declaration of time 2017-02-25 Fabio M. Di Nitto [transports] add option to configure reconnect interval 2017-02-24 Fabio M. Di Nitto [hb] use savederrno correctly 2017-02-23 Fabio M. Di Nitto [tests] knet_bench add support for perf-by-time [tests] add first cut of perf-by-size implementation to knet_bench Also port to _sendmmsg/_recvmmsg and fix a couple of minor shutdown issues 2017-02-22 Fabio M. Di Nitto [tests] don't spam for no logs [transport] fix sendmmsg and recvmmsg wrappers to return correct data for sendmmsg also don't overwrite the msg_len 2017-02-21 Fabio M. Di Nitto [libknet] drop bad doc copy/paste Merge pull request #29 from liu4480/master add doc for knet_handle_get_channel in libknet/libknet.h 2017-02-21 Bin Liu add doc for knet_handle_get_channel in libknet/libknet.h 2017-02-21 Fabio M. Di Nitto [build] add better header protection fixes: https://github.com/fabbione/kronosnet/issues/23 [build] Enforce C99 compile option fixes: https://github.com/fabbione/kronosnet/pull/25 [sctp] fix more occurrences of link -> kn_link complete 940833bf99b18e14b08c0a5a24402026131a0884 fixes: https://github.com/fabbione/kronosnet/issues/28 [build] don't relink the whole test suite on each change fixes: https://github.com/fabbione/kronosnet/issues/27 2017-02-17 Fabio M. Di Nitto Merge pull request #21 from liu4480/master Fix: remove duplicate judgement of (have_name < 0) in kronosnetd/vty_cli_cmds.c 2017-02-17 Bin Liu Fix: remove duplicate judgement of (have_name < 0) in kronosnetd/vty_cli_cmds.c In line 1087 of kronosnetd/vty_cli_cmds.c, there is a judgement of (have_name < 0), when have_name < 0, it will return 0, which make the judge of have_name < 0 in line 1094 will never be executed. Remove the unnecessary judgement. 2017-02-15 Fabio M. Di Nitto [tests] port knet_bench to use internal version recvmmsg [global] drop usage of recvmmsg inside libknet as requested by different kernel developers, we should stop using both sendmmsg and recvmmsg. as temporary solution use Jan's compat wrappers. the whole TX/RX code will need review to do a full proper switch since all error codes will change and propagate differently to transport hooks and knet_send_sync users [global] drop usage of sendmmsg inside libknet as requested by different kernel developers, we should stop using both sendmmsg and recvmmsg. as temporary solution use Jan's compat wrappers. the whole TX/RX code will need review to do a full proper switch since all error codes will change and propagate differently to transport hooks and knet_send_sync users [global] create our own copy of mmsghdr in preparation to drop sendmmsg and recvmmsg knet_bench will drop mmsghdr when dropping sendmmsg and recvmmsg [poc] drop PoC sctp-defrag-bug code now that's merged into master 2017-02-14 Fabio M. Di Nitto [udp] remember to memset freshly allocated memory [sctp] drop duplicated definition [global] revert b525ee6a6b74ec, go back to 64K nodes, and use typedef for nodeid [tx] increase timeres on TX pressure and reduce log noise when TX sockets are overloaded, we spend more time spitting out logs than recovering from the overload. ifdef the logging on critical path out (still available with debug build). also drastically reduce the waiting time by 64x. this changes increases UDP perf on 3 nodes by 200% [tx] pass pointers around and save some memory [tx] minor optimization in handling packets < MTU - reduce of one loop - reduce one memcpy - reduce setting a bunch of data [tx] map any outgoing packets only once and reduce memsets around [tx] map outgoing packet once per host vs once per host per link - reduces memsets by 8x - reduces complexity in resending packets on socket overload by changing start mmsghdr vs remapping everything all over again 2017-02-07 Fabio M. Di Nitto [transport] define value only once 2017-02-06 Fabio M. Di Nitto [tx] fix error log 2017-02-05 Fabio M. Di Nitto Move TODO list to trello 2017-02-03 Fabio M. Di Nitto [global] Reduce the number of support nodes in a knet from 64K to 256 realistically speaking nobody is ever going to deploy more than 32/64 nodes. NOTE: this commit changes both onwire and API! 2017-02-02 Fabio M. Di Nitto [send/recv] split RX and TX threads in separate files [send/recv] split code into separate files (step 1) [send/recv] Unify and simplify usage of seq_num in packets IMPORTANT: this commit changes onwire protocol in an incompatible way! - remove the concet of bcast and mcast seq num and use one tx_seq_num - stop using LINK_UP_DOWN messages to broadcast node seq num and transfer this data inside heartbeat messages - LINK_UP_DOWN messages are currently unused but let's keep the infrastructure around for future - minor cleanup in host_set_policy to confirm change of switching policy in the logs - _link_updown should use async call to host dstcache update due to locking context - switch knet_link_set_enable to use write locking context since the only reason it was read lock, was due to the need to send LINK_UP_DOWN messages - knet_link_set_priority can now use dstcache in sync mode - add seq_num and heartbeat type (timed/untimed) data to heartbeat messages. timed messages are generated regularly by hb_thread. untimed messages are generated by the TX thread to sync seq_num on heavy load across all connected node. (see comments in the code) - access to the node seq_num is now mutex locked - abstract ability to send pings from multiple threads - special case seq_num == 0 to detect a node crash and coming back to life before hb_thread can detect the disconnection - forcefully send ping in the TX thread every SEQ_MAX / 8 packets to allow nodes to sync seq_num - optimize TX thread code to prepare the outgoing buffers once vs multiple times. There is still work that can be done here to optimize sending to multiple host, but this change is intrusive enough already as it is - add logic to clear circular buffers when receiving pings [sctp] Allow internal IPC to operate under RX pressure 2017-01-28 Fabio M. Di Nitto [rpm] disable build of poc code 2017-01-27 Christine Caulfield [compat] Fix FreeBSD defines We need to undefine HAVE_RECVMMSG & HAVE_SENDMMSG for it to compile on FreeBSD (not sure how that ever worked before). There is a PR for FreeBSD to fix the recvmmsg bug, but it's not in a released update yet. So with luck we can remove this soon-ish https://github.com/freebsd/freebsd/commit/3fa64907439c00965fb0cbb222a6bf073c452630 2017-01-27 Fabio M. Di Nitto [host] remove completely broken host to host communication locking system the original idea was to have a host-to-host (semi-)reliable communication protocol but that just isn't possible without flow control and retransmit IIRC the only side affect of this missing lock is a corner case where: 1) node A totally crashes 2) node A come backs to life, sends it's status info (seq_num information) 3) node B does NOT receive the status info 4) node A starts sending traffic and a few packets might get lost this will be solved when rewriting the TX thread to optimize the seq_num handling [send/recv] make sure to unlock dsthost thread under certain errors [transport] make sctp tx error handling more robust and avoid spinning on dead sockets Update TODO [build] add build option to enable/disable poc code 2017-01-26 Fabio M. Di Nitto [transport] fix support for dynamic links connections - add internal transport API for handling incoming dynamic connections (both UDP and SCTP) - fix copy/compare address code in RX thread - make sure to reset sockaddr_storage len in iov [netutils] cleanup cmpaddr and add cpyaddrport cpyaddrport specifically copies _only_ address and port and skip over other fields that are unnecessary for what we need 2017-01-26 Christine Caulfield compat: make it work on FreeBSD Although FreeBSD 11 has sendmmsg & recvmmsg wrappers, they don't quite work the same as Linux so I've enabled the (fixed) compat versions for that platform. make check now works on FreeBSD 11 2017-01-26 Fabio M. Di Nitto [kronosnetd] add support for multiple transports 2017-01-25 Fabio M. Di Nitto Update TODO [send/recv] simplify _parse_recv_from_links internal API [transports] add support for transport_tx_sock_error add IP_RECVERR for UDP to better handle errors on TX sockets [PMTUd] Fix global pmtud calculation 2017-01-24 Fabio M. Di Nitto Update TODO list Merge pull request #20 from jonesmz/travis-ci-for-pull-request Adds Travis CI continuous integration configuration file [transports] SCTP: handle short reads on sockets - fix trasport is_data API - handle per accepted socket reassembly buffer by changing fd_tracker data for incoming connections - allow in-kernel SCTP fragmentation again - use MSG_EOR on a per socket base to reassemble partial packet delivery - fix some whitespaces around 2017-01-24 Michael Jones Adds Travis CI continuous integration configuration file 2017-01-24 Christine Caulfield transport: Rename 'link' to 'kn_link' so it doesn't conflict with libc 2017-01-23 Christine Caulfield poc: add option to check CRCs in sctp_defrag_bug 2017-01-20 Fabio M. Di Nitto [PoC] implement EOR handling code for short reads 2017-01-19 Fabio M. Di Nitto [poc] add sctp potential fragmentation bug trigger [build] drop last bits of common/ [transport] remove duplicate code 2017-01-18 Fabio M. Di Nitto Update TODO [transport] add knet_handle_get_transport_id_by_name api [transport] add knet_handle_get_transport_name_by_id api call also fix a counter in knet_handle_get_transport_name_by_id [transport] add knet_handle_get_transport_list api call [handle] fix error messages 2017-01-17 Christine Caulfield [compat] Fix some includes for FreeBSD 2017-01-17 Fabio M. Di Nitto [compat] minor cleanup on the autotool stuff 2017-01-17 Christine Caulfield general: Add compatibility code & #ifdefs This is enough to get knet compiling on FreeBSD 11 and bits of it working. It's nowhere near ready on BSD though, more work is needed, but given the fast pace of development it's best to get this in now rather than track it in a separate branch. 2017-01-17 Fabio M. Di Nitto Update TODO Update TODO 2017-01-16 Fabio M. Di Nitto [api] add commodity functions to convert to/from strings/sockaddress functions are nothing more than wrappers for getnameinfo and getaddrinfo with some sanity checks, but exposing them around saves lots of maintanance of duplicate code across different stuff. 2017-01-16 Christine Caulfield host: Remove host_tail as it's not used. it also had a bug which could cause knet to crash after a host was removed. This patch puts new nodes at the front of the list rather than at the end, but I don't think this is a problem. 2017-01-16 Fabio M. Di Nitto [test] fix comment [sctp] enable workaround for in-kernel fragmentation issue [test] kill obsoleted ping_test Update TODO [sctp] port sctp to the new API and fix many issues - use the new transport API - fix locking context around to avoid race conditions and deadlocking - fix shutdown code (segfaults and core dumps) - properly differentiate between connecting sockets and accepted sockets and better use of fd_tracker - abstract as much as possible socket management code from threads - add lots of comments and debugging messages around - simplify socket error management reported from RX thread - rework loop timers for connect_thread to avoid thread overload - reduce usage of _transport_addrtostr to one call and reuse link information around - improve error handling across the board - stop using data structs inside epolls and switch back to using fds this was necessary to avoid processing stale data from epolls and look up data from fd_tracket instead - add listener stop function - make functions safer to be called multiple times - probably more.. but can't remember NOTE: this is not the most elegant code, but it seems to be doing its job fine. [heartbeat] reset PMTUd on link down to trigger a new PMTUd when link come back online [PMTUd] fail early if transport is not connected [heartbeat] and make it a tiny little bit faster when transport is not connected [heartbeat] make better use of transport_connected option 2017-01-15 Fabio M. Di Nitto [sockets] move socketpair code into transport common code also use _configure_common_socket for cloexec/nonblock and rx/tx buffers that should be the same for all socketpairs, outgoing and incoming sockets [dstthread] make sure not to get caught in signal when reading from socket [host] make sure not to get caught in signal when sending host updates 2017-01-12 Fabio M. Di Nitto [transports] drop fd_tracker locking in favour of global locking [udp] drop unnecessary list locking 2017-01-11 Fabio M. Di Nitto [SCTP] temporary disable SCTP support while we fix the implementation not the best solution, but it's a quick one to allow master to build again [send/recv] report correct errno Revert "[heartbeat] temporary disable usage of transport_connected" This reverts commit 75ca10991fa3e0d9bc8cd9706395ff0e2bc62c38. whops.. pushed in the wrong branch by mistake [heartbeat] temporary disable usage of transport_connected this will make it easier to stabilize SCTP and connection oriented protocols 2017-01-09 Fabio M. Di Nitto Update TODO list [transports] major rework of APIs and ABIs! This commit breaks APIs and ABIs and this changelog might be missing a bit or 5. External visible changes: - Change link initialization process API: The old method: - link_set_config -> link_enable where link_set_config would simply store config data (sockaddr and such) in the link struct and link_enable would create sockets and do the magic. This method didn't work well for complex transports such as SCTP and introduced a series of race conditions and deadlocks. The new method: - link_set_config would now store config in link struct AND create all related sockets and such. The link will not be used for data traffic till enabled. - link_set_enable will enable/disable the link for traffic (including heartbeat) and requires a link to be configured (as before) - link_clear_config (new API) can/has to be called after disabling a link to close all connections and sockets (free resources). Internal changes: - Drop the concept of listeners.* and delegate those to underlying transports. - Add the concept of fd_tracker. Each transport is required and it is responsible to update the fd_tracking array for the fd that the transport itself is creating/using/clearing. The fd_tracker is required to perform fast lookups on socket errors and RX thread to determine what code is responsible to parse given conditions such as errors from the sockets or OOB data/notifications. - Introduce the concept of link->transport_connected. In case the transport requires socket connection to the other end (SCTP for example), set to 0 while disconnected and set 1 once connected to the other side. This flag will avoid unnecessary errors generated from the TX threads. NOTE: still needs better plumbing around. For now it's only partially used in the heartbeat thread. UDP sets to 1 by default. - Rework the transport API to be easier to use. - If a transport is not available, get_XXXX_transport() should return NULL. - A transport that provides &XXXX_transport_ops MUST have all operations implemented as described in internals.h. This is required to skip tons of if/else checks on fast code paths. - Improve documentation of the transport API in internals.h. - transport common: drop some unnecessary functions for now, they might can back later in better format once SCTP is working again. - provide a locked/unlocked version of _set_fd_tracker but this is an artifact of trying to fix SCTP deadlocking. _set_fd_tracker should always be locked. - cleanup transport_udp.c to match the new API and perform better error handling and better cleaning in case of errors. - switch mtu_overhead to constant from call into a function. - drop usage of _transport_addrtostr/_transport_addrtostr_free for all but accepting incoming SCTP connections. All other datas are already available and it's unnecessary to perform extra lookups. This will eventually move to a proper knet_api and avoid completely the need to build with --debug. - RX thread: - use the new transport hooks to handle socket errors and determine if a packet is data or internal OOB info. - remove the last transport specific bits and move them into transport implementation itself. - Test suite: - Update the test suite to deal with the new API changes - Fix a couple of shutdown problems in knet_bench - Fix a shutdown issue in test-common.c - Document the whole recvmmsg API in details, based on kernel implementation. recvmmsg man page is incorrect. - Fix kronosnetd to use the new APIs 2016-12-21 Fabio M. Di Nitto [send/recv] remove stray unlock operation 2016-12-20 Fabio M. Di Nitto [logging] rename KNET_SUB_SCTP_LINK_T to KNET_SUB_TRANSP_SCTP [logging] rename KNET_SUB_UDP_LINK_T to KNET_SUB_TRANSP_UDP [logging] rename KNET_SUB_LINK_T to KNET_SUB_RX [logging] rename KNET_SUB_SEND_T to KNET_SUB_TX [logging] rename KNET_SUB_PMTUD_T to KNET_SUB_PMTUD [logging] rename KNET_SUB_HB_T to KNET_SUB_HEARTBEAT [logging] rename KNET_SUB_TRANSPORT_T to KNET_SUB_TRANSPORT [logging] rename KNET_SUB_SWITCH_T to KNET_SUB_DSTCACHE [logging] assign blocks to different logging subsystems - drop KNET_SUB_PMTUD that was unused - add KNET_SUB_TRANSPORT_T that was missing - switch from "common" to "unknown" for odd logging - fix up code around to allow holes in the structs - fix up test suite 2016-12-19 Fabio M. Di Nitto [PMTUd] adjust all values around new transports [build] cleanup conditionals to build SCTP and other protocols Unsupported protocols just need to return NULL on get_PROTO_transport and contain all protocol conditionals within the transport_PROTO.c. There is still one conditional in RX/TX thread that needs cleaning but it's a rather complex operation and leave it in for now. [transports] initialize transports with knet_handle and protects from missing proto ops [listener] drop unnecessary debugging code 2016-12-18 Fabio M. Di Nitto [sctp] _handle_incoming_sctp don't leak accepted_socks on errors add a check to verify max connections or we will leak FDs somewhere at some point [logging] threads should have _t naming 2016-12-17 Fabio M. Di Nitto [sctp] don't pass notification packets to the decoder 2016-12-16 Fabio M. Di Nitto [build] make SCTP build optional (default: on) 2016-12-16 Christine Caulfield sctp: Fix a couple of crashes & tidy on_epoll usage Also fix locking in sctp shutdown 2016-12-16 Fabio M. Di Nitto [build] don't leak pam libraries all over the place [links] fix segfault when using dynamic ip addresses Update TODO list [build] cleanup configure.ac and Makefile.am - check for libqb only if building kronosnetd - check for SCTP headers and fails if we don't find them - export DEBUG conditional in Makefiles - conditionally include netutils.c for DEBUG [rpm] fix BuildRequires for SCTP [kronosnetd] Fix build after SCTP merge Default to UDP for now, add support for multiple protocols later on [libknet] fix typo 2016-12-15 Fabio M. Di Nitto Update TODO list Merge pull request #18 from chrissie-c/topic-cc-transports Factor out protocol code and add new ones 2016-12-13 Christine Caulfield internals: Finish renaming QB_LISt macros to KNET_LIST libknet: Fix things raised in review 2016-12-02 Christine Caulfield libknet:add TCP transport libknet: add SCTP transport libknet: Add support for multiple transports Changes the API to add transport to knet_link_set_config & knet_link_get_config libknet: Factor out UDP into its own file With a transport_common file for protocol-independant things. 2016-11-25 Fabio M. Di Nitto [poc] remove iov-enc code now merged in master [tests] Add knet_bench as more flexible replacement for ping_test [send/recv] Fix bug in handling HOSTINFO packets seq num the bcast/ucast seq num were being mixed up, causing packet loss on link up/down (also in active mode). 2016-11-21 Fabio M. Di Nitto [tests] add facility to stop a whole knet handle in one shot [tests] allow scheduler to be configured and allow start_logging to be called multiple times safely also test crypto_bench with multiple schedulers 2016-11-19 Fabio M. Di Nitto [tests] use SCHED_RR for crypto_bench 2016-11-16 Fabio M. Di Nitto [tests] add options to configure scheduler [tests] move crypto bench to use start_logging [tests] add knet logs: prefix when no logs are being recorded [tests] add warning on when to use or not to use start_logging [tests] Fix Makefile.am to build check_PROGRAMS [tests] add logging thread support to simplify writing tests 2016-11-15 Fabio M. Di Nitto Merge pull request #16 from miz-take/prototype saved errno when epoll_ctl failed 2016-11-15 Takeshi MIZUTA saved errno when epoll_ctl failed 2016-11-15 Fabio M. Di Nitto Update TODO list [tests] rename fun_crypto_benchmark to crypto_bench and create a benchmark section in tests/Makefile.am [tests] fix up input handling on crypto tests [crypto] drop the unnecessary hash_block code this was a leftover from corosync port of the code and it was a failsafe for buffer allocation apparently. buffer allocation was completely reworked in corosync to dynamically account for headers and so on. this also removes one memcpy and shave some more time in hashing operations. [tests] dynamically allocate memory for int_crypto_test and make it easier to extend text to some input boundaries. also make check-memcheck doesn't appear to spot bad memory access on allocated-on-stack buffers [test] move benchmark outside of regular testing [crypto] cleanup some headers and drop duplicate code [crypto] introduce (nss)crypto_encrypt_and_signv and move crypto_encrypt_and_sign to use the v version. [poc] add io-hashing PoC [tests] fix hello world! test 2016-11-07 Fabio M. Di Nitto [tests] Add basic functional crypto benchmark test 2016-11-06 Fabio M. Di Nitto [tests] add internal crypto basic test Fix build with different gcc Integrate it all with build Simplify build of poc-code Simplify even further. EncContext tracks the location of the buffer and doesn't care about tmp_len fix PoC to handle buffer data alignment 2016-11-05 Fabio M. Di Nitto Add libnss iovec PoC this is a small modified version of the code example available here: https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/nss_sample_code/NSS_Sample_Code_sample2 to show that it is possible to encrypt from multiple src buffers into a final buffer. this approach will allow us to remove a lot of memcpy around the recv/send thread and replace them with iovec mapping all across. Possible side effects: - less memory consumption (mapping vs allocating) - more complex code / crypto (internal) API changes Add poc-code directory and move access-list code from Chrissie there Over time we have created a lot of small PoC code to make knet better. Most of it is already integrated but other chunks are still floating around. I prefer to keep them here so that: 1) they don't get lost 2) it will be easier to integrate them once we get to that point 3) we can share PoC and work on them together Add notes for package maintainers 2016-10-22 Fabio M. Di Nitto [build] use rpm bcond for a more elegant solution to override ./configure defaults [build] fix spec/srpm/rpm build build only packages that we are configuring for (--enable-libtap, --enable-kronosnetd) and have spec file depends on the latest configure run to regenerate the info 2016-10-21 Fabio M. Di Nitto [build] fix make distcheck to include kronostned/libtap coverage [build] fix conditional build to pass make distcheck [tests] fix return code checking spotted by gcc 5.3.1 Merge pull request #15 from chrissie-c/master config: Make libtap & kronosnetd optional 2016-10-21 Christine Caulfield Make sure we build libtap if kronosnetd is also being built I didn't think it worth checking for and using local libtap, if anybody else wants that I will take patches. config: Make libtap & kronosnetd optional 2016-10-19 Fabio M. Di Nitto Merge pull request #14 from jfriesse/include-compat-h [libknet] Really use compat when needed 2016-10-19 Jan Friesse [libknet] Really use compat when needed Previous patch added compatatibility layer for systems without (recv|send)mmsg but sadly forgot include compat.h when needed. 2016-10-18 Fabio M. Di Nitto Merge pull request #13 from jfriesse/genericmmsg [libknet] Support systems without (send|recv)mmsg 2016-10-18 Jan Friesse [libknet] Support systems without (send|recv)mmsg Implemented are generic (slightly reduced functionality) functions and for systems with syscall available but libc prototypes unavailable also syscall wrapper. 2016-10-15 Fabio M. Di Nitto [send/recv] err is never <0 in that code path [listener] fix listener sock configuration to match other sockets add send buffer and avoid EAGAIN issues on high load by delegating buffering to the kernel 2016-10-13 Fabio M. Di Nitto Merge pull request #12 from jfriesse/lib64 [build] Make sure lib64 is used when available Merge pull request #11 from jfriesse/donotoverridesyncindex [libknet] Do not override symbols Merge pull request #10 from jfriesse/removeprlimit [libknet] Use getrlimit instead of prlimit 2016-10-13 Jan Friesse [build] Make sure lib64 is used when available Default location for libraries is PREFIX/usr/lib. This doesn't work well on systems with mixed 64 and 32-bit libs. [libknet] Do not override symbols sync and index are both stdlib symbols so it's better not to use them as function arguments. [libknet] Use getrlimit instead of prlimit prlimit is Linux specific. Because passed pid is 0, we can replace prlimit call by portable getrlimit without loose of functionality. 2016-09-15 Fabio M. Di Nitto [tests] Enable pmtud setfreq test in helgrind context update helgrind suppression file [PMTUd] rewrite thread to follow the same footsteps of heartbeat thread drop all those notification mechanisms that are a locking nightmare with very little to none gain and use a loop similar to heartbeat. Also break down the code in 3 sections: - main thread loop - pmtud data gathering per link - pmtud data gathering for all hosts/links [threads] move thread timer resolution to common section 2016-09-13 Fabio M. Di Nitto [tests] move all API checks makefile bits to a separate file prepare for int(ernal)_ and fun(ctional)_ tests [tests] drop normal info and leave only progress/warnings/errors [tests] Add knet_handle_enable_pmtud_notify tests [tests] remove unnecessary host_remove [tests] Add knet_handle_pmtud_get tests [PMTUd] Fix knet_handle_pmtud_get API sanity checks [tests] Add knet_handle_pmtud_getfreq tests [PMTUd] Fix getfreq API check Revert "[tests] enable memcheck and helgrind checks by default" This reverts commit 7d3ca6729c224c6a592a71ba8813f3d5b811912c. [tests] enable memcheck and helgrind checks by default 2016-09-11 Fabio M. Di Nitto [tests] add api_knet_handle_pmtud_setfreq tests currently not enabled for helgrind checks. PMTUd thread needs a good rewrite first but the API is fine. [tests] expand knet_send for basic send checks [tests] add api_knet_send_sync tests and update helgrind suppression file [send/recv] Fix checks for knet_send_sync and move sync test earlier [tests] print warnings for tests with exception enabled [tests] export check type to all tests This allows to skip/warn on some tests while we fix the PMTUd thread 2016-09-06 Fabio M. Di Nitto [tests] remove stray line 2016-09-04 Fabio M. Di Nitto [tests] add knet_send tests [handle] Allow threads to recognized shutdown before stopping them. On busy threads and shutdown, there is a chance that the thread didn't have time to process the shutdown requests and even tho we are closing down, some locks can be destroyed while still locked. In theory we could have each thread report that it has ack'ed the shutdown and then close, but a sleep seems to be more than good enough for now. [send/recv] reworking locking on send to link protect reading of datafd when processing outgoing data and avoid unnecessary double locking in _handle_send_to_links (calling functions both uses locking already). [handle] expand knet_send input sanity checks [tests] fix ping_test to deal with knet_recv API fixes [tests] fix memcpm test and typos 2016-09-03 Fabio M. Di Nitto [tests] Add knet_recv tests [handle] fix checks around knet_recv 2016-08-28 Fabio M. Di Nitto [tests] add knet_handle_get_datafd tests [tests] add knet_handle_get_channel tests [tests] add knet_handle_remove_datafd tests [tests] add knet_handle_add_datafd tests [tests] drop suprious cleanup entry [tests] add knet_handle_enable_sock_notify tests [tests] add knet_handle_enable_filter tests 2016-08-26 Fabio M. Di Nitto [tests] add knet_host_enable_status_change_notify tests [tests] add knet_host_get_status tests 2016-08-21 Fabio M. Di Nitto [tests] add knet_link_get_status tests [tests] re-order test executions based on class [tests] align coding style and error checking [tests] add knet_handle_setfwd tests 2016-08-17 Fabio M. Di Nitto [crypto] fix nss init/shutdown memory leaks make nss_init more robust across invokations [build] separate memcheck and helgrind suppressions files 2016-08-13 Fabio M. Di Nitto [tests] add knet_handle_crypto tests [crypto] fix knet_handle_crypto config checks remove redundant and duplicated code from nsscrypto 2016-08-12 Fabio M. Di Nitto [tests] fix out-of-tree path to valgrind suppression file [build] make clang happy [crypto] drop redudant checks spotted by clang [build] make clang happy [handle] Fix check for malloc result spotted by clang [build] fix configure.ac to deal with clang return codes being weird when testing for an option alone clang will return a warning and RC=0 unless -Werror is also specified, and that warning switch to RC=1. [tests] fix missing check for api_knet_link_get_priority [tests] add knet_link_get_link_list tests [tests] add knet_link_get_enable tests and fix a potential segfault [tests] fixup knet_link_set_enable tests add disable code checking and silence a helgrind warning. helgrind complains about the heartbeat code writing back to the link status vs knet_host_add. knet_host_add is using an wr lock, heartbeat code a read lock. the heartbeat code is the only one changing two ping timers values. nothing else in the code is using/changing them. knet_host_add does a memset at alloc time. tho, it's impossible to hit the HB code _before_ or _during_ knet_host_add, therefor it's safe to silence the warning. 2016-08-11 Fabio M. Di Nitto [tests] add knet_link_set_enable tests 2016-08-10 Fabio M. Di Nitto [tests] add knet_link_get_priority tests 2016-08-09 Fabio M. Di Nitto [tests] add knet_link_set_priority tests 2016-08-05 Fabio M. Di Nitto [tests] add progress message 2016-07-02 Fabio M. Di Nitto [build] chose between the lesser of 2 evils [tests] rename all tests to add postfix _test 2016-07-01 Fabio M. Di Nitto [link] rename knet_link_get_timeout to knet_link_get_ping_timers and add related tests Also in the process, make sure to init timers when configuring a link and export link defaults in libknet.h [link] rename knet_link_set_timeout to knet_link_set_ping_timers and related tests [tests] add knet_link_get_pong_count tests [tests] fix knet_link_set_pong_count check [tests] Add knet_link_set_pong_count tests [tests] remove unused variable [link] fix knet_link_get_config API and add related tests 2016-06-30 Fabio M. Di Nitto [libtap] make testing of symbols more consistent with libknet [build] be consistent in building libraries and use same export standards [tests] make it consistent and easier to run check-memcheck and check-helgrind 2016-06-29 Fabio M. Di Nitto [tests] add api_knet_link_set_config tests 2016-06-27 Fabio M. Di Nitto [tests] don't sort output and use header order to display API calls makes it easier to follow the logic 2016-06-26 Fabio M. Di Nitto [tests] extend api test coverage to match header and binary exported symbols [libknet] exports only public API symbols [tests] fix api_knet_log_set_loglevel incorrect exit sequence [tests] Add knet_log_get_loglevel tests [tests] add knet_log_set_loglevel tests 2016-06-25 Fabio M. Di Nitto [build] add comment [logging] fix knet_log_get_loglevel_id crash, return code and add related tests [logging] fix knet_log_get_loglevel_name crash, return correct values and add related tests [logging] Fix knet_log_get_subsystem_name crash, make call consistent and add related tests [logging] fix knet_log_get_subsystem_id crash and add related tests 2016-06-24 Fabio M. Di Nitto [host] fix possible race condition in accessing host data bonus: remove a lock used on all threads all the time cons: traffic can pause more often towards a host if another host links are not stable [host] fix knet_host_get_policy API and add related tests [host] Fix knet_host_set_policy API and add tests [tests] add knet_host_get_host_list test [host] fix knet_host_get_id_by_host_name API and add related tests [host] fix knet_host_get_name_by_host_id API and add related tests 2016-06-21 Fabio M. Di Nitto [tests] add simple tool to report api testing coverage 2016-06-20 Fabio M. Di Nitto [host] return an error on names that are too long vs truncating [tests] add api_knet_host_set_name test [host] fix knet_host_set_name api checks verify that we actually get a name from the caller and make sure name is unique or search would break 2016-06-19 Fabio M. Di Nitto [tests] add knet valgrind suppression file remove some error reporting for now, since they are known and harmless race conditions. [tests] make ping test run under valgrind and init all memory correctly [send/recv] make sure to initialize memory properly [send/recv] fix invalid memory access when checking channel values 2016-06-18 Fabio M. Di Nitto [tests] add api_knet_host_remove test 2016-06-17 Fabio M. Di Nitto [global] move netutils to common dir (take 2) [host] fix build failure [global] move netutils to common dir those will be handy for libknet testing [tests] add api_knet_host_add test [host] perform check before locking [tests] add api_knet_handle_free tests [logging] rework locking around logging make sure to check/fill data after locking and complete write before release the lock. also use trylock vs lock since most log are called while already locked and it's cleaner. [handle] drop redundant check in knet_handle_free If there are no hosts configured, there are also no listeners configured [tests] make it clear where log is coming from and avoid confusion from logs generated from our test suite [tests] fix some exit paths to flush logs and close fds 2016-06-16 Fabio M. Di Nitto [handle] drop redundant check now the the whole process is locked at global level [tests] add api_knet_handle_new test [PMTUd] Allow thread to have time to react to shutdown before cancelling the thread and solves an issue deleting locks (spotted by helgrind) on exit [handle] cleanup socketpair creation for good Better version of 1f890b897474352a376b27cfe79365237791d66f Don't accept fd values higher than max fd allowed by the OS 2016-06-14 Fabio M. Di Nitto [threads] fix locking across the board for shutdown 2016-06-13 Fabio M. Di Nitto [libtap] make sure memory is init properly (according to valgrind) [tests] make helgrind errors a failure [handle] be consistent in applying sockopts around [handle] fix fd leak and sock option settings apparently int math has never been my strong side [tests] fix running tests with valgrin under libtool Update gitignore [tests] fix special check include Revert "[tests] move all tests to top level directory" This reverts commit d9adee9b923776ea5813b5a024c16a58b8d989c0. [tests] move all tests to top level directory [tests] fix libtap test [test] Add common check for root user knet tests can't really run as normal user [tests] add some common facility to run shell codes within tests code stolen from libtap, but useful to drive iptables and alike during some tests Fix core dump on exit path when running as user and related test case. Drop the max host to 1024 for testing. [tests] add common return codes as understood by automake 2016-06-12 Fabio M. Di Nitto Move libknet tests in their subdir in preparation for proper test suite [PMTUd] export protocol overhead information 2016-06-11 Fabio M. Di Nitto Update TODO list [PMTUd] Drop concept of link_mtu and fix data mtu exported values (pass 1) Cleanup a poor attempt to export all MTU data to the users. The original idea was to provide information about real link onwire mtu and max user data mtu (that could be sent on wire without fragmentation) that adjusted the value for IP protocol, knet header and crypto (if configured). Effectively there is no use for the real link onwire mtu and it only involves extra complexity to export lots of extra data to the end user that would have to redo the calculations internally to gather data mtu. Also fix the status.mtu value to match knet view of every link. 2016-06-03 Fabio M. Di Nitto Update TODO list Update TODO list [PMTUd] fix link instability due to too short PMTUd timeouts Partially roll back changes from 004bd26a 2016-06-01 Fabio M. Di Nitto Update TODO [socket] internal socketpair should use same recvbuf as any other socket 2016-05-16 Fabio M. Di Nitto Merge pull request #4 from mbaldessari/arm-fix Initialize savederrno variable 2016-05-15 Michele Baldessari Initialize savederrno variable This fixes an error seen on F23 arm/gcc: In file included from threads_send_recv.c:26:0: threads_send_recv.c: In function '_handle_send_to_links_thread': logging.h:19:2: error: 'savederrno' may be used uninitialized in this function [-Werror=maybe-uninitialized] log_msg(knet_h, subsys, KNET_LOG_ERR, fmt, ##args) ^ threads_send_recv.c:475:6: note: 'savederrno' was declared here int savederrno, docallback = 0; ^ cc1: all warnings being treated as errors Makefile:843: recipe for target 'libknet_la-threads_send_recv.lo' failed make[2]: *** [libknet_la-threads_send_recv.lo] Error 1 make[2]: Leaving directory '/home/jenkins/workspace/kronosnet/arm-fedora-23-gcc/libknet' Makefile:506: recipe for target 'all-recursive' failed make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory '/home/jenkins/workspace/kronosnet/arm-fedora-23-gcc' Makefile:417: recipe for target 'all' failed make: *** [all] Error 2 2016-03-06 Fabio M. Di Nitto [PMTUd] fix several issues around PMTUd and improve discovery performances - Fix a race condition around first time MTU discovery where MTU value would be out of the wazooo - Fix a case where onwire MTU value would be smaller than crypto header and generate errors - Make MTU value a decision factor if a link is usable or not. If MTU is not valid (either the discovery process failed) or the value is not sane, the link will be removed from the available data link pool till the problem is resolved. Once the problem is solved the link will recover automatically. - Have link up/down status trigger a PMTUd process to speed up detection. - Add ability to detect smaller MTU than minimum defined by protocol. - Improve error reporting across the board. - Make PMTUd timeout dynamic based on current link latency. This change makes detection of small MTU very fast. - Sanity check values returned by PMTUD to be in range between minimum MTU defined by given protocol and maximum that knet can handle. - Trigger host cache reculaculation on MTU validity change to reduce risk of packet loss on bad links. 2016-02-16 Fabio M. Di Nitto [send/recv] fix handling of duplicated HOSTINFO pckt with active-active links [internal] rename list_rwlock to global_rwlock 2016-02-15 Fabio M. Di Nitto [host] fix memory leak in knet_host_add in case of failure make it easier to read knet_host_remove (no functional changes) [internal] document usage of mutex and locks Update TODO list Update TODO list 2016-02-11 Fabio M. Di Nitto Update TODO 2016-02-10 Michele Baldessari [docs] Fix docbook typo 2016-02-10 Fabio M. Di Nitto [crypto] fix crash when receiving non crypto packets on a crypto socket [onwire] let's try to be smart and preallocate space for future features 2016-02-09 Michele Baldessari Merge pull request #3 from mbaldessari/wip_overflow_fedora Fix small string overflow Fix small string overflow In file included from /usr/include/stdio.h:936:0, from ping_test.c:12: In function ‘snprintf’, inlined from ‘main’ at ping_test.c:504:3: /usr/include/bits/stdio2.h:64:10: error: call to __builtin___snprintf_chk will always overflow destination buffer [-Werror] return __builtin___snprintf_chk (__s, __n, __USE_FORTIFY_LEVEL - 1, Merge pull request #2 from mbaldessari/smatch_fixes Assorted fixes found via smatch Fix another indentation issue Fix equality test Fix broken indentations Move len to ssize_t len is used as a return code for both select()(int) and read()(ssize_t) so make it an ssize_t type instead of size_t which is unsigned. Fix error path test equality 2016-02-09 Fabio M. Di Nitto Update TODO list 2016-02-07 Fabio M. Di Nitto Update TODO list 2016-02-06 Fabio M. Di Nitto [send/recv] add knet_send_sync function read carefully the doc in libknet.h or suffer the ethernal doom in hell 2016-01-10 Fabio M. Di Nitto [sockets] first cut at making socket management more consistent and fix sendmmsg handling 2016-01-05 Fabio M. Di Nitto Update TODO list 2015-12-24 Fabio M. Di Nitto Update TODO 2015-12-20 Fabio M. Di Nitto [send/recv] rework handling of 0 byte read/write on local sockets considering there is no obvious use case trying to send 0 byte packets and that 0 byte packtes are filtered by iovec calls across the all code, there is no point trying to handle 0 byte differently from any other socket error. this commit makes sure that every time there is an error (-1) or a 0 byte read from the locally provided sockets, a call back is issued. on read errors (-1) the socket will be removed from the epoll to avoid spinning and it is safe to call knet_handle_remove_datafd afterwards to remove it completely. it is now mandatory to enable a sock_notify callback before adding datafd. read libknet.h carefully on what the callback is supposed to do based on the type of socket your application is adding. 2015-12-18 Fabio M. Di Nitto Update TODO list remove stray comment 2015-12-18 Christine Caulfield [lib] Fix typos & English in libknet.h file 2015-12-18 Fabio M. Di Nitto [send/recv] deal properly with 0 bytes packets on local sockets [send/recv] be consistent between readv and recvmmsg handling 0 bytes packets [PMTUd] improve debug message to add iface mtu [global] add support for multiple local sockets/fds WARNING: this commit changes API and onwire protocol It is now possible to configure multiple local sockets to send/recv data over the same knet handle. to every send/recv socket a channel is assigned that behaves more or less like a VLAN. hostA channel 0 will be delivered to datafd on hostB channel 0 hostA channel 1 to hostB channel 1 etc. It is possible to configure up to 32 channel, but there is space to increase this number if necessary. New API calls: knet_handle_add_datafd knet_handle_remove_datafd knet_handle_enable_sock_notify knet_handle_get_channel knet_handle_get_datafd Notification has been added in cases where a local socket is the result of accepting a tcp connection that gets disconnected and corrective action needs to be taken by the application. Changed API calls: knet_handle_new knet_recv knet_send knet_handle_enable_filter onwire changes: add one byte to transport channel information channel information are also passed down to the dst_host_filter in the event the application has needs to modify that on TX/RX events. Internal changes: cleanup how sockpairs are handled in general to make it simpler to map application side and internal side. NOTE: callback function has not been properly tested. 2015-12-16 Fabio M. Di Nitto Update TODO list [internal] switch from pipe to socketpairs for consistency [send/recv] add support for recvmmsg on local socket Update TODO list 2015-12-14 Fabio M. Di Nitto Update TODO 2015-12-13 Fabio M. Di Nitto [send/recv] save some more times dealing with unreachable hosts for mcast pckts [send/recv] rework commit 1de7acb8 a bit on the exit thread, kh_node is knet_h->host_id, no point to convert it N times per packet/frag. [send/recv] drop too verbose log message and fix comment [send/recv] remove TODO comment splitting frags across multiple links is not worth the complexity of the code or the complexity in re-assemblying the packet if links are at different speed/status [send/recv] sort code around Update TODO list [send/recv] fix bcast data entry for hostinfo packets this should address the issue Chrissie reported where first data packet would vanish in void 2015-12-12 Fabio M. Di Nitto [send/recv] drop redundant check [send/recv] fix logging levels Update configure.ac for sendmmsg [host] cleanup circular buffer function names Sort TODO list [host] add host status change callback notifications API is also ready for further expansion (external and remote support), but they are currently not implemented. allow both notifications and polling of status via knet_host_get_status use new notification and api polling in ping_test.c as unit test use the new host status information in the TX thread to skip some time consuming pckt processing if the destination host(s) is not reachable 2015-12-11 Fabio M. Di Nitto [send/recv] fix data len passed to dst host filter on the RX end 2015-12-10 Fabio M. Di Nitto [filters] be consistent and provide private data for all notifications/callbacks 2015-12-09 Fabio M. Di Nitto [send/recv] improve dst_host_filter API to include more data to determine destination [send/recv] add support for dynamic buffer allocation for packet re-assembly this commit changes the on-wire protocol. 2015-12-08 Fabio M. Di Nitto [send/recv] fix host_id value in host_filter thanks Chrissie for reporting the problem 2015-11-30 Fabio M. Di Nitto [send/recv] Fix handling of HOSTINFO packets HOSTINFO packets are nothing more than special cased DATA packets. The contents of HOSTINFO is stored inside DATA of DATA packets and up to the point where we need to use the HOSTINFO data, we need to treat the packets as DATA for defragmentation and deduplication. This also fixes a problem when using round-robin and active switching policy where the HOSTINFO exchange would loop forever in a storm because duplicate pckts were being handled at once. 2015-11-29 Fabio M. Di Nitto Update TODO list 2015-11-28 Fabio M. Di Nitto Replace memcpy with memmove across the board memmove is just a tiny littlebit faster than memcpy. Update TODO list Update TODO [PMTUd] Fix crash and calculation when using crypto most of the problem was around incorrect typing of variables and incorrect calculation when using signing without encryption. 2015-11-27 Fabio M. Di Nitto [libtap] automatic restore IPv6 address when recovering from MTU flapping NOTE: if a user manually changes MTU < 1280, your IPv6 addresses are gone for good. Update TODO list 2015-11-26 Fabio M. Di Nitto Update TODO list 2015-11-25 Fabio M. Di Nitto [crypto] align crypto_authenticate_and_decrypt API to crypto_encrypt_and_sign 2 issues solved by this change: 1) API were different and crypto_authenticate_and_decrypt would trash the incoming packet, that we might want to keep for later usage (re-switch for example) 2) By using an extre pre-allocated buffer while decrypting incoming packets we save a whole memcpy and that reduces latency on crypto communications: pre-patch: [root@kronosnet-node1-br0 ~]# ping 192.168.12.2 -f -c 1000 -s 65000 PING 192.168.12.2 (192.168.12.2) 65000(65028) bytes of data. --- 192.168.12.2 ping statistics --- 1000 packets transmitted, 1000 received, 0% packet loss, time 5302ms rtt min/avg/max/mdev = 5.102/5.283/22.679/0.947 ms, pipe 3, ipg/ewma 5.307/5.223 ms post-patch: [root@kronosnet-node1-br0 ~]# ping 192.168.12.2 -f -c 1000 -s 65000 PING 192.168.12.2 (192.168.12.2) 65000(65028) bytes of data. --- 192.168.12.2 ping statistics --- 1000 packets transmitted, 1000 received, 0% packet loss, time 5196ms rtt min/avg/max/mdev = 4.997/5.154/6.661/0.129 ms, ipg/ewma 5.201/5.173 ms (using aes256 and sha1) 2015-11-22 Fabio M. Di Nitto [send/recv] init recv buffer array only once [internal] define/use PCKT_FRAG_MAX instead of UINT8_MAX makes the code more readable Fix error message [send/recv] use recvmmsg on network sockets On my test systems this change reduces the latency of fragmented packets from 0.27ms to 0.235ms in average. There are still some optimizations on how buffers can be initialized that could shave some time on this code path. Update TODO 2015-11-18 Fabio M. Di Nitto Update copyright dates [build] update configure.ac 2015-11-16 Fabio M. Di Nitto [send/recv] fix data filling in the frag code [send/recv] use sendmmsg to reduce latency On my test systems this change reduces the latency from 0.7ms to 0.25ms when sending 65000 bytes packet and using internal re-assembly code. Internally use N buffers to split the incoming data and reduce by N frags the amount of memcpy between kernel and userland. [internal] reduce memory footprint further down 2015-10-24 Fabio M. Di Nitto [global] reduce memory footprint by a long shot [send/recv] add simplistic packet re-assembly code this will probably explode if you are using multiple links in round-robin or active-active mode. 2015-10-21 Fabio M. Di Nitto Revert "[global] reduce memory footprint by a long shot" This reverts commit 4b4290a2fafbeb4d08bf489caa35ea4c205b18b9. 2015-09-20 Fabio M. Di Nitto [global] reduce memory footprint by a long shot [threads] cleanup header [cleanup] remove unused headers around [threads] split send recv threads in separate file [threads] split dsthandler thread in separate file [threads] split pmtud thread in separate file [threads] split heartbeat thread in separate file [threads] implement outgoing pckt fragmentation based on PMTUd 2015-09-16 Fabio M. Di Nitto Update TODO list [global] restrict max packet size to 65536 bytes and apply workaround in data buf allocation while we cleanup the whole buffer allocation handling situation 2015-09-15 Fabio M. Di Nitto [test] disable crypto rekey test for now [knetd] fix libqb logging filter Thanks to Chrissie for the awesome work as usual [test] fix ping_test data print and add some more info [handle] add missing include [handle] fix knet_send sock handling to use correct data flow [handle] fix init order to have more debug in case of sockpair error [commodity] add knet_send/knet_recv api calls based on readv/writev man page, it is highly discouraged to mix different kind of write/read operations on a given fd (for example a read with a writev). since not many finds using iovec that pleasent, those 2 functions do wrap the more common buff/buff_len calls into iovec implementations. [handle] API change warning: add support for knet_handle_new to create proper sockpairs this is a small facility for applications that don't want/need to create socketpairs to read/write data to/from knet and avoids lots of duplicated socket management on the client side. [threads] add warning on fragmented packets also prepare ping_test for it 2015-09-08 Fabio M. Di Nitto [ping] fix sockpair creation to deal with sequence of packets and not streaming [threads] use iovec calls to read/write from datafd [cosmetic] rename net_fd to datafd 2015-09-06 Fabio M. Di Nitto [handle] add missing errno on failure [libtap] change error check code to avoid a segfault and make it more readable 2015-04-21 Fabio M. Di Nitto [listener] fix 2 cosmetic checks and keep around a useful debugging fn [threads] make ping error message more verbose Update TODO list 2015-04-21 Christine Caulfield libtap: don't close pipe fd twice libtap was closing the parent's pipe() fd twice. Sometimes, between the first and second close, that fd gets reopened by the logging thread as PF_LOCAL, and used for logging. Then libtap wakes up again after the fork and closes the fd thinking it's its pipe fd. knet then opens it again as AF_INET for pinging. libqb logging then errors on a sendto() and closes it AGAIN, and reopens it as a PF_LOCAL logging fd. Ping then has a brain haemorrage next time it comes round to calling sendto(). 2015-03-20 Fabio M. Di Nitto Update TODO list 2015-03-19 Fabio M. Di Nitto [host] move dst cache updates to the right logging subsystem and return error codes Update TODO list [host] create dst_cache sync and async functions and fix link lookup race condition Prior to this change, there was a potential race condition in which, when shutting down a link, the destination cache would not be updated in time and pckts would be lost in an invalid socket. This change introduced a sync version of the dst cache update and protects outgoing traffic socket lookup with a mutex. The only downside of this change is that, in future, when we will introduce complex switching algorithms, we might have to spend a bit more time recalculating the network topology on link shutdown and halt traffic switching. Pro is that we have a smaller gap in lost traffic when reconfiguring the local node. 2015-03-18 Fabio M. Di Nitto [threads] simplify code to send data to links in preparation to remove race condition in _dst_cache update, simplify the access path to common code [onwire] fix regression in macro expansions 2015-03-12 Fabio M. Di Nitto [onwire] introduce khpi_link_status_ defines to make code simpler to parse 2015-03-11 Fabio M. Di Nitto [onwire] add define for knet_hostinfo structures size 2015-03-10 Fabio M. Di Nitto [onwire] rename kf_plink/kf_psize/kf_pdata to their respective more descriptive versions [onwire] rename kf_link and kf_time to their respective more descriptive versions [onwire] rename kf_seq_num and kf_data to their respective more descriptive versions [onwire] add comment [onwire] more cleanup around _SIZE define usage 2015-03-09 Fabio M. Di Nitto [onwire] rename various KNET_FRAME_ definitions to KNET_HEADER_ [onwire] cleanup and use defines make it easier to understand values and usage of other status info [onwire] rename link_up_down struct to knet_hostinfo_payload_link_status [onwire] rename knet_hinfo_data struct to knet_hostinfo and add comments around the various fields 2015-03-08 Fabio M. Di Nitto [onwire] add comment in ifout section [onwire] start cleaning host_info data structures [debug] add tool to print out network pckt header size [onwire] clenaup knet_header_payload_data structure naming [onwire] clenaup knet_header_payload_ping structure naming 2015-03-07 Fabio M. Di Nitto [cleanup] move some values from int to unsigned int [onwire] clenaup knet_header_payload_pmtud structure naming and add more comments [onwire] add more comments [onwire] add extra comments [onwire] cleanup knet_header_payload union use longer name to make it easier to identify nesting of structures [onwire] rename knet_frame_data to knet_header_payload [onwire] rename knet_frame to knet_header also start renaming variables around to make it easier to understand the level of nesting and add comments for each field [internal] start renaming things to make code look decent [internals] Simplify code maintanance by shuffling around some variables this allows creation of better #define to maintain some parts of the code related to buffers allocations. As a bonus, gcc does a better job at packing structs outside of a union and we save 4 bytes onwire. 2015-03-05 Fabio M. Di Nitto [knetd] Fix internal MTU calculation to include tap ethernet frame 2015-03-04 Fabio M. Di Nitto [PMTUd] fix mutex deadlock on socket error [listeners] Use IP_FREEBIND options on sockets this allows us to bind to IPs that are not yet on the system and survive the infamous ifdown/ifup test 2015-03-03 Fabio M. Di Nitto [pingd] fix ping flood on sock errors if we get an error on sendto, we would ping flood the link because the ping_last timestamp was not being updated. [libknet] rename tap to send to link plain and simple cleanup 2015-03-02 Fabio M. Di Nitto Update TODO list [knetd] add support for pmtud frequency [PMUTd] add call to get pmtud interval value [knetd] Use pmtud values to set tap device MTU 2015-02-28 Fabio M. Di Nitto [PMTUd] fix notification callback to add private data this is necessary to identify the callback when driving multiple knet instances within the same daemon. Update TODO list 2014-12-01 Fabio M. Di Nitto [PMTUd] add support for pmtud_get for who doesn't want notifications [PMTUd] add support for PMTUd changes notifications 2014-11-30 Fabio M. Di Nitto [PMTUd] distinguish between link_mtu and data_mtu link_mtu is the pure MTU over the link and doesn't account for knet headers. data_mtu is what can be really transmitted onwire by the application. 2014-10-23 Fabio M. Di Nitto [PMTUd] add support for timer config and fix critical shutdown deadlock in condwait [PMTUD] change error message [PMTUd] fix scary message at startup 2014-10-21 Fabio M. Di Nitto [PMTUD] fix threads to work with crypto temporary disable re-keying test as it seems to be break PMTUd 2014-10-20 Fabio M. Di Nitto [PMTUD] change variable names to make it easier to read still doesn't work with crypto enable, but part of the code is getting ready for it. 2014-08-29 Fabio M. Di Nitto [nss] fix calculation for crypto header to include SALT_SIZE and propagate block size up in the layers to allow PMTUd thread to round buffers and do crypted PMTUd packets. 2014-08-28 Fabio M. Di Nitto [link] add rudimentary PMTUD support. 1) It won't work with crypto enable yet 2) values are not really used yet for anything 3) no API calls implemented yet 2014-08-11 Fabio M. Di Nitto [listener] check setsockopts return codes 2014-07-23 Fabio M. Di Nitto [nss] make sure that any crypto config requires a proper hashing value this is mandatory to avoid some collision checks when random junk is received over the network socket 2014-07-21 Fabio M. Di Nitto [libtap] check for error buffers only when necessary [build] fix pkg-config files to link correctly [build] libtap requires sys/types.h for size_t definition 2014-07-20 Fabio M. Di Nitto [build] cleanup after test-driver update .gitignore and make maintainer-clean target 2013-11-16 Michele Baldessari This fixes build on F20 with gcc 4.8.2 2013-08-31 Fabio M. Di Nitto Update slides 2013-06-15 Fabio M. Di Nitto libknet: add extra error checking 2013-06-10 Fabio M. Di Nitto kronosnetd: add support for pong_count link: add pong count handling to avoid excessive flapping of traffic 2013-06-09 Fabio M. Di Nitto cleanup TODO list crypto: track size of crypto overhead data libknet: comment out un-used onwire data for now 2013-03-11 Fabio M. Di Nitto Update TODO list 2013-03-10 Fabio M. Di Nitto build: fix install target listener: fix more unistd.h breakage 2013-02-01 Fabio M. Di Nitto Every set should have a get... cleanup a few bits around logging and interal API 2013-01-30 Fabio M. Di Nitto libknet: keep host list cached in knet handle Update TODO list 2013-01-29 Fabio M. Di Nitto libknet: add link api call to get list of configured links Finish to cleanup current logging API logging: start documenting API and other bits logging: fix formatting fix README pam section Fix ping_test to deal with EINVAL return codes Update TODO list libknet: fix knet_link_get_status and document it kronosnetd: deal with proper error codes from knet_link_get calls libknet: fix set/get_priority and document them libknet: fix set/get_timeout and document them Add knet_link_get_enable call ping_test: some more fixes how the hell did this thing ever worked? needs a rewrite link: fix return code when exit path is clean 2013-01-28 Fabio M. Di Nitto fix \n logging listener: rename link to lnk to avoid conflict with unistd.h ping_test: fix argc corruption and make output configurable threads: make one error more specific libknet: rework knet_link_enable into knet_link_set_enable make it thread safe and fix all return codes (ret vs errno) 2013-01-27 Fabio M. Di Nitto Update TODO list 2013-01-20 Fabio M. Di Nitto link: protect set/get config from doing damages link: rename configured to enabled we need configured internally for other stuff 2013-01-11 Fabio M. Di Nitto libknet: cleanup knet_link_set/get_config this also allows to kill the madness around dynamic ip management and saves enother bytes in the ping header Fix make dist/clean 2013-01-10 Fabio M. Di Nitto Fix typo debian: fix a few bits around and resync with rpm packaging Update TODO list Update TODO list 2013-01-08 Fabio M. Di Nitto systemd: allow kronosnetd to set real time scheduler see also https://bugzilla.redhat.com/show_bug.cgi?id=893015 Update TODO list logging: move string to the beginning of msg this allow (to some extents) to print log messages directly to stdout and/or stderr without extra parsing. NOTE: - messages are sent from differnet threads and there for they are less readable when they arrive all together - there is no printing of message level and subsystem for simple coding/debugging purposes this is useful, but otherwise look at how ping_test.c or kronosnetd create a logpipe and read sequencial messages from there. Update TODO list rewrite TODO list to be more useful 2013-01-07 Fabio M. Di Nitto libknet: fix HMAC key init 2013-01-06 Fabio M. Di Nitto start cleaning TODO list and make it more useful Sort TODO list 2013-01-02 Fabio M. Di Nitto Update TODO list 2013-01-01 Fabio M. Di Nitto Update TODO list 2012-12-31 Fabio M. Di Nitto pam: use password-auth on systems like fedora that ship it or passwd as a fallback (debian and others) rpm: fix integration with systemd (missing sysconfig file) and pam/groupadd rpm: fix systemd integration with spec file 2012-12-25 Fabio M. Di Nitto kronosnetd: fix integration with fedora pam 2012-12-24 Fabio M. Di Nitto Update TODO list Downgrade error from set scheduler to warning Fix build dependencies for parallel build Add missing header to final tarball 2012-12-21 Fabio M. Di Nitto Start documenting knet_link_* api 2012-12-20 Fabio M. Di Nitto Update TODO list fix crash when multiple links are sharing the same listener Fix shutdown race conditions between locks/threads/signals that in turns make knet_handle_free thread safe Fix log subsystem on _dst_cache_update make knet_host_(get|set)_policy behave as documented also fix _dst_cache_update to return errno and use host_id Update TODO list make knet_host_get_host_list work as documented in the API make knet_host_get_id_by_host_name work as documented in the API make knet_host_get_name_by_host_id work as documented in the API Fix pthread error to errno relation make knet_host_set_name comply with documented API make knet_host_remove comply with documented API also fix a bunch of other bits make knet_host_add comply with documented API also continue renaming node_id to host_id as we go handle: fix errno return value 2012-12-19 Fabio M. Di Nitto Make knet_handle_* call compliant to documented API allow knet_handle_crypto to be invoked at runtime libknet: make knet_handle_new / knet_handle_free do what the API says they should move timespec_diff to threads.h (only user) Make define name clear Cleanup internal headers Move _dst_cache_update to host function set and maybe add the threads.* files to git might help.. just a tiny little bit Move all threads code into threads.{c,h} Move all link related code to link.{c,h} Move listener function definitions in listener.h move logging code/definitions to logging.{c,h} move host.c function definitions into host.h move some definitions out of internals.h into common.h Start renaming node_id to host_id Document knet_handle_crypto document a couple of more knet_handle_* calls Update TODO list 2012-12-18 Fabio M. Di Nitto libknet: document knet_handle_enable_filter and dst_host_filter_fn One more cosmetic libknet: cosmetic fixes to knet_handle_new libknet: knet_handle_new API cleanup and documentation libknet: document limits move comment where it belongs libknet: make dst_filter a runtime config option libknet doesn't need a default port it's a UI issue to deal with Reduce logging buffer size to allow more messages Add comments to knet_link and re-arrange move knet_link into internals.h hack fest still in place, we need to sort out ABI and locking 2012-12-17 Fabio M. Di Nitto libknet api cleanup: move knet_host out of public API the knet_host_* calls are pretty much set for now. the new knet_link_* calls are temporary to interface with the old API. knet_link will die next. Move --version to library only bits Update TODO list 2012-12-14 Fabio M. Di Nitto reorganize libknet.h in sections just makes it easier to read as we move stuff around Drop tests/ subfolder Move all tests in the respective folders this makes it easier to isolate what needs changing where and when Move hellgrind and valgrind checks to a common make snippet Cleanup tests for kronosnetd remove obsoleted test tool Drop one more unrequired config bit move all logging definitions in the same section drop redundant dst_filter config check fix segfault in ping_test src_port handling move listener only define cleanup crypto headers move ping size to onwire and make name more clear drop unrequired header libknet: create onwire.h to contain/define all packet(s) data/info Make rename clean rename libknet-private.h to internals.h 2012-12-13 Fabio M. Di Nitto Move etherfilter where it belongs 2012-12-07 Fabio M. Di Nitto libknet: fix host info propagation on kronosnetd restart this avoids lots of incorrect chit/chat and make sure to send data when required Update TODO list 2012-12-06 Fabio M. Di Nitto kronosnetd: fix init script return codes 2012-12-04 Fabio M. Di Nitto One more attempt to fix make dist out-of-tree better fix to fix git fix for build out of tree fix Fix out-of-tree build Update TODO list fix O_NONBLOCK fd setting functions *facepalm* logging: cleanup logging to avoid clogging the logfd pipe move it to its own thread and clean up logfd handling Fix debian generation debian: fix date spacing/formatting libtap: small step back to .so.0.0.0 no point to have 1.0.0 when we are not even sure the API is final build: fix maintainer-clean when building on != debian based distro Update .gitignore libknet: fix data forwarding check Update TODO list 2012-12-03 Fabio M. Di Nitto Update TODO list build: fix make dist and make maintainer-clean add support to build deb packages use: make debs 2012-12-02 Fabio M. Di Nitto generate package version from git/configure 2012-11-29 Fabio M. Di Nitto Update TODO list Fix copyright/authors all over kronostned: add logrotate file update TODO list Fix super annoying kronosnet/kronosnetd discrepancies around and fix a few rpmlint warnings/errors Update .gitignore kronosnetd: add basic spec file add native debian packaging this is based on debhelper version 7 to allow build also on Ubuntu Lucid update from autoscan kronosnetd: add man pages init script: shutup lintian warning 2012-11-26 Fabio M. Di Nitto libknet: fix cbuffers cleaning after normal shutdown/link down/crash 2012-11-25 Fabio M. Di Nitto Update TODO list libknet: clean up circular buffer counters when remote goes dead libknet: record status from incoming HOST_INFO link up-down messages 2012-11-23 Fabio M. Di Nitto libknet: notify remote host of link status changes receiver still can't use those data for anything libknet: fix locking within logging... for real what the hell was i thinking with 1ab701ce16 libknet: fix locking within logging we cannot grab a lock during logging because something else might be already using a lock. libknet: add support for HOST_INFO unicast dispatch libknet: fix data write to fd Update TODO 2012-11-17 Fabio M. Di Nitto libknet: remove hardcoded magic numbers libknet: basic prototyping of HOST_INFO messages libknet: make all data packed gcc doesn't propagate packed attribute down to a union libknet: fix unlocking in case of errors in tap_to_links 2012-11-16 Fabio M. Di Nitto Update TODO list libknet: add sync host to host messagging kronosnetd: make baseport endian clean kronosnet: merge concept of RINGID and baseport this avoids extra confusion with ringid 2012-11-14 Fabio M. Di Nitto Update TODO list libtap: store ip in the same order as they entered kronosnetd: save interfaces in the same order as input kronosnetd: allow vty timeout to be loaded from config Update TODO list 2012-11-13 Fabio M. Di Nitto Update TODO list libtap: attempt to fix ioctl usage kronosnetd: fix parsing of ip_list return from libtap 2012-11-12 Fabio M. Di Nitto Update TODO list (bug list) kronosnetd: increase default vty timeout and save it to config loading still has no effect kronosnetd: add vty timeout command the value is not persistent or saved in config kronosnetd: add vty config node Update TODO list cannot be done without major config rewrite right now kronosnetd: fix spacing in login prompt Update TODO list 2012-11-10 Fabio M. Di Nitto Update TODO list 2012-11-09 Fabio M. Di Nitto libknet: fix link_enable logic 2012-11-08 Fabio M. Di Nitto kronosnetd: perform clean shutdown on daemon exit libknet: fix listener remove return code libknet: misc cleanup remove fds from epolls on exit don't mingle with listeners if there is no link status change libknet: fix unlocking in log message libknet: make basic infrastructure ready for host to host comm libknet: rename pipefd to dstpipefd Update TODO list 2012-11-06 Fabio M. Di Nitto crypto: move crypto buffer allocation back to handle nsscrypto doesn't need to know how many buffers to allocate/free libknet: drop magic number in header save 4 bytes per packet and avoid plain text attack Update TODO libknet: make error message more useful kronosnetd: make config safer to copy paste kronosnetd: add ringid to interface creation to spicy up mac generation fix multihoming support major change across the board: - drop concept of listeners completely, they are now handled internally - link needs src/dst - fix different segfaults - cleanup API - port kronosnetd UI to new format 2012-11-02 Fabio M. Di Nitto libknet: first big hammer on logging still needs fine tuning as we go to understand exactly what to log in debugging mode kronosnet: use a different macro for debug from libknet Update TODO list kronosnetd: add logging from libknet to logfile/stdout no logging to vty yet 2012-11-01 Fabio M. Di Nitto libknet: add basic logging infrastructure to the library 2012-10-28 Fabio M. Di Nitto Update TODO list Rename config fd to make space for logging from knet 2012-10-27 Fabio M. Di Nitto Update TODO list for libknet logging infrastructure Update TODO list Using multiple dynamic links requires: link dynamicXXXXXXXX where XXXXXXXX can be any identifier Use bindv6only by default and make code simpler and safer 2012-10-26 Fabio M. Di Nitto Update TODO list libknet: fix bindv6only socket handling for dynamic connections 2012-10-25 Fabio M. Di Nitto libknet/kronosnetd: add support for bindv6only kronosnetd: fix error message and exit path kronosnetd: add support for dual stack CLI (bindv6only yes/no) kronosnetd: increase CLI default timeout libknet: fix timespec handling guess what.. it ain't portable... Update TODO Update TODO list 2012-10-24 Fabio M. Di Nitto kronosnetd: don't buffer overflow update .gitignore build: link as needed general: add init script and service file libknet: ship .pc file and sort linking libknet: link against -lrt for clock_gettime symbol build: fix build for old version of nss/nspr build: add missing file from tarball Add preliminary support for handling dynamic ip connections max on dynamic link per host for now, we need to fix link_id handling first 2012-10-23 Fabio M. Di Nitto kronosnetd: provide basic status functionality ping_test: fix unit measure to be bearable ping_test: fix usage example kronosnet: add config support for link timers kronosnetd: add link priority configuration libknet: add link priority api call 2012-10-15 Fabio M. Di Nitto libtap: workaround bug in SETMTU ioctl apparently SETMTU does parse flags as well, make it clean kronosnetd: allow configuration of switching-policy engine passive/active/round-robin update TODO list 2012-10-14 Fabio M. Di Nitto libknet: worst commit ever.... fix a series of segfaults in crypto init code add api call to set switching policy (that was the original idea of this change, till things started exploding on me) libknet: readd dst_cache_update function for internal use this is required to trigger a recalculation when we change switching policy at runtime from active/passive to active/active or round-robin libknet: don't use black magic.. libknet: simplify link updown notification code libknet: remove that horrid dst_cache update and mask it around link_enabled makes the API a bit more readable. 2012-10-13 Fabio M. Di Nitto libknet: fix 3des key size libknet: add support for 3des nss crypto libknet: add support for nss aes128 and aes192 Update TODO list 2012-10-12 Fabio M. Di Nitto libknet: move crypto to fully modular system libknet: start implementing generic crypto vs nsscrypto framework it will save us some time later if we decide to plug something else crypto: rework the API and make it a separate init call this also saves us a few memcpy around when not using crypto 2012-10-07 Fabio M. Di Nitto libknet: make host add a O(1) operation 2012-10-06 Fabio M. Di Nitto libknet: keep host ordered by insert kronosnetd: drop trailing space when saving config libknet: fix memmove offset and make it simpler libknet: first cut for RR support per packet 2012-10-05 Fabio M. Di Nitto libknet: add handling of link priority for active passive also update the TODO list (this commit has not been tested) update TODO list based on rekey brainstorm 2012-10-03 Fabio M. Di Nitto kronosnetd: use fixed mac addresses to reduce arp work libknet: export function to allow dst cache update on link removal update TODO list link policy handler: add first cut of code to have a switching policy engine for now only ACTIVE and PASSIVE are supported. RR needs dst_cache_update/lookup. 2012-10-02 Fabio M. Di Nitto libknet: rename var to be more explicit libknet: fix endianess and memory alignment issues also check for some return codes build: drop warning check for enum-compare in some older versions of gcc, the flag is supported only for C++, our flag test succeed, but we fail to build. 2012-10-01 Fabio M. Di Nitto libknet: add unicast switching support 2012-09-30 Fabio M. Di Nitto update TODO with some ideas for improvements 2012-09-29 Fabio M. Di Nitto build: LIBQB -> libqb libknet: rename fields in knet_link to be more meaningful libknet: rename i,j,k to some proper vars and make code a bit more readable libknet: make ready for unicast handling and fix unlocking if crypto fails update TODO list 2012-09-28 Fabio M. Di Nitto libknet: packet inspector first cut add packet destination filter API and provide an example for ethernet packet inspector. also start to plug the inspector into the transmitting end of libknet libknet: implement basic ring buffer for bcast packets fix KNET_FRAME_SIZE handling around don't try to send messages to links that are not ready gather host info for all incoming pkcts (data/ping) since there is no point to go crazy figuring it out add basic ring buffer functions based on playground implementation (still missing reset of the buffer) kronosnetd: fix data initialization we can't memset knet_link because khost_add does some initialization there that we overwrite otherwise. drop incorrect clean up path. knet_links are not malloc'ed but statically allocated. 2012-09-27 Fabio M. Di Nitto kronosnetd: remove spurios debugging messages libknet: add seq_num to sender host bcast for packets that needs to be sent to all nodes ucast for direct packets the difference is necessary to avoid unncessary re-encryption of packets. This will make more sense once packet inspector is in place, but basically seq_num are per host and not all packets will go everywhere, creating undesirable holes in the packet_ring_buffer, that in turns, in a non-encrypted network could lead to some annoyance. libknet: change packet definitions to include seq_num for data packets this is required for pckt deduplication kronosnetd: node_id have to match knet size for packet inspector to work 2012-09-26 Fabio M. Di Nitto libknet: add notes on packet inspector and where it should hook for notifications build: fix out of tree build clean up some hardcoded values and export them via header correctly kronosnetd: add support for encryption keys kronosnetd: add knet-keygen 2012-09-25 Fabio M. Di Nitto fix a few int vs ssize_t bits around vty: add partial crypto/hash support libknet: fix nss types and add crypto/hash support to ping_test 2012-09-24 Fabio M. Di Nitto libknet: plug nss for good into threads libknet: crypto simplify even more the code path and data storage libknet: add macro for debugging of nsscrypto and simplify more code around libknet: more crypto cleanup and allocate crypto buffers when necessary libknet: add a few comments libknet: more nss crypto work make crypto_init/fini work. make libknet shared (necessary for correct linking with nss). 2012-09-21 Fabio M. Di Nitto libknet: start plugging in crypto support 2012-09-20 Fabio M. Di Nitto libknet: make tap to links and links to tap full duplex this cuts down by 40% latency on ping test ping_test: clarify unit measure... 2012-09-17 Fabio M. Di Nitto libknet: add totemcrypto.c|h from corosync (pristine) permission to copy and change licence to GPLv2+ has been granted in writing by current authors of the code. let's be more realistic on what we can do here libvty is not going to happen. Too complex and it's not the goal for this project. For now, let's move it back into the daemon. 2012-09-16 Fabio M. Di Nitto logging: add logging support for syslog/stderr/logfile hook into libqb for logging facilities. build: cleanup escaping in exporting vars 2012-09-15 Fabio M. Di Nitto drop generic utils.(c|h) and start implementing proper logging logging: add concept of logfile and fix fprintf output build: depends on nss and libqb build: move daemon in its own dir libtap: remove external header (useless really) 2012-09-14 Fabio M. Di Nitto libtap: mark library 1.0.0 drop fPIC (managed correctly by libtool) and make it 1.0.0 (number vs version) build: fix some minor bits around 2011-03-23 Fabio M. Di Nitto main: print errors on stderr before daemonizing testsuite: decouple from utils.c libtap: decouple test suite from utils.h cfg: drop util.h vty: add missing include and fix build decouple cloexec calls from utils test: rename test to match what it really does move netutils in libvty/ since vty is the only user Revert "libknet: add missing include" This reverts commit c5edcc14952da3e9914f1ff8df98413ab4728c2c. logging: fix utils_syslog handling set default to syslog to 0 again and enable it only in kronosnetd, otherwise all test suite would log to syslog too. utils: remove timespec_diff that's relevant only to libknet libknet: add missing include utils: drop concept of STATIC vs static vty: make more functions static 2011-03-22 Fabio M. Di Nitto logging: enable syslog by default also inject log_debug into syslog if configured main: move defaults into its own function Move vty into its own dir for future "librarification" Move libtap doc to the next release fix file permission 2011-03-09 Fabio M. Di Nitto Add some extra warnings at build time 2011-01-24 Fabio M. Di Nitto Update TODO: API renames are done build: use libtap vs libtap_test 2011-01-19 Fabio M. Di Nitto libtap: embed test suite in the library libtap: move tests in libtap/ update TODO for libtap 2011-01-18 Fabio M. Di Nitto libtap: fix linking and set exported symbols 2011-01-05 Fabio M. Di Nitto main: remove oom_adj kernel ABI will change, not really worth the troubles for now 2011-01-03 Fabio M. Di Nitto libtap: align API calls to correct names libtap: improve error reporting for up/down operations 2010-12-23 Fabio M. Di Nitto libtap: propagate errors from CLI calls to upper layers 2010-12-22 Federico Simoncelli libknet: use naming convention for static functions misc: more file renaming libknet: split code into multiple files libknet: update naming convention, rename header files 2010-12-22 Fabio M. Di Nitto libtap: first pass to remove calls to log_* use internal error buffer instead across the non public API. pass2 will export it to users. 2010-12-21 Federico Simoncelli knet: move knet code to its own library 2010-12-20 Federico Simoncelli ring: fix node_id endianness, use ping/pong packet mask ring: use hosts index to send pings and receive pongs 2010-12-20 Fabio M. Di Nitto vty_cli_cmds: bind pre-up/up/down/post-down libtap: add updown tests and fix off by one check libtap: add exec_updown wrapper for ifup/down operations Update TODO libtap: several improvments across the board start to add pre-up/up/down/post-down support cleanup internal API function names fix memory leak on close add forward declearation for all static functions add an extra test to be executed under valgrind test: add track-fds to valgrind check 2010-12-18 Fabio M. Di Nitto Update TODO list libtap: complete ip_list management and last integration with CLI libtap: make libtap more thread safe 2010-12-17 Fabio M. Di Nitto build: clean some more vty_cli_cmds: prepare for new knet_handle_new API Merge branch 'master' of github.com:fabbione/kronosnet Update TODO libtap: move ip management into libtap 2010-12-17 Federico Simoncelli utils: fix timespec diff on x86 platforms 2010-12-17 Fabio M. Di Nitto libtap: rename internal struct from knet_tap to tap_ libtap: fold interface name into libtap and avoid dup use libtap: add knet_tap_find, switch to ifname macro also revert a0cbd79cefdbfed1d41f8842d6786f5b55484aa4 to fix unnamed tap device creation add more TODO entries vty_cli_cmds/cfg: drop local knowledge of MTU and delegate to libtap libtap: add reset to default functions for mtu/mac TODO list update for libtap libtap: do proper initialization and close of the library store default mtu and mac at init. This operation is required also for direct access to set_mtu and set_mac. libtap: cleanup tap socket on last close 2010-12-16 Federico Simoncelli Merge branch 'master' of github.com:fabbione/kronosnet todo: update todo list 2010-12-16 Fabio M. Di Nitto libtap: start introducing internal list/config tracking libtap: remove unrequired strcpy vty: remove debugging entry libtap: switch to use opaque struct build: I HATE MAKEFILE EXPANSION.... fix libtap build kthxbye build: link test tools with shared libtap do not link kronosnetd yet. build: need two versions for each lib test and final. build: include libtap subdir in the build libtap: rename knet_test to tap_test libtap: add shared version of the library NOT TO BE USED YET libtap: align API to name convention First cut to isolate tap handling in its own library vty_cli_cmds: port VTY to new API needs lot more work to get clean vty_cli_cmds: port "interface" to new ring API 2010-12-10 Federico Simoncelli knet: new api design 2010-12-03 Federico Simoncelli ring: improve looping over links 2010-12-02 Federico Simoncelli ring: allocate links with host 2010-11-30 Federico Simoncelli ring: fix major bugs in knet frame length ring: add host index for fast lookup tests: fix long integer syntax 2010-11-29 Federico Simoncelli ring: improve knet_frame structure utils: fix timespec difference 2010-11-27 Fabio M. Di Nitto knet: make internal functions static TODO: remove redundant entry TODO: add more items for shared libs build: add .bz2 tarball to release build: fix uninstall section move kronosnet.conf to /etc/kronosnet/kronosnet.conf add more todo items 2010-11-26 Fabio M. Di Nitto update TODO Add TODO list and release planning docs: add architecture doc Add README file 2010-11-25 Fabio M. Di Nitto vty/cfg: remove two unrequired wrappers vty: remove bridge struct vty: destroy knet_h on error and "no interface" 2010-11-24 Federico Simoncelli git: add benchmarks to ignored files benchmark: add lookup benchmark utils: improve knet_ts_diff speed Merge branch 'master' of github.com:fabbione/kronosnet knet: add sigint handler 2010-11-24 Fabio M. Di Nitto cfg: remove unused struct 2010-11-23 Federico Simoncelli knet_handle: stop heartbt thread on knet_handle_free knet_handle: check pthread exit status knet_handle: add knet_handle_free knet_handle: do not generate new sockets for knet_handle 2010-11-23 Fabio M. Di Nitto vty: add link menu and clean up links on object destroy path vty: save bridge status in config file ring: avoid lots of unnecessary casting, fix build on sparc and fix link loop also avoid a segfault when scanning for multiple links 2010-11-22 Fabio M. Di Nitto vty: add start/stop commands to interface configure interface kronosnet0 2 baseport 50000 mtu 9000 ip 192.168.7.2 24 peer node1 1 link 192.168.5.1 link 192.168.4.1 exit start exit exit exit now starts the bridge and traffic is flowing! ring: fix address comparison to determine if link is up/down also initialize addrlen properly and fix make dist build: update configure.ac after autoscan build: fix make dist vty: remove concept of auto-listeners/manual listeners switch to a much simpler baseport and port range concept. for now, only baseport is implementend and needs extra checks implementation. portrange will come soon enough. ring: fix ping calculation for good finally I have been able to reproduce the problem and it's related to the "unknown" source in MONOTONIC clock. It can rollover making the signed comparison useless. Force unsigned long long for the calculation that fixes the issue. Switching from: Notice: timediff: -345483470 interval: 1000000 To: Notice: timediff: 18446744073387515858 interval: 1000000 ring: use long long instead of suseconds_t it appears (but not totally sure) that suseconds_t is not easily portable. On sparc it was showing, at random, some different behaviour. 2010-11-19 Fabio M. Di Nitto Merge branch 'master' of github.com:fabbione/kronosnet vty: fix no link list handling 2010-11-19 Federico Simoncelli Merge branch 'master' of github.com:fabbione/kronosnet vty: add 'O' to pre-escape sequences 2010-11-19 Fabio M. Di Nitto vty: initialize timeout values vty: add basic link support build: fix sparc build 2010-11-19 Federico Simoncelli tests: fix printf format for latency tests: display errno for functions supposed to fail knet: fix maximum socket receive buffer size (unit is bytes) Merge branch 'master' of github.com:fabbione/kronosnet knet: add listener test cases 2010-11-19 Fabio M. Di Nitto vty: get autolisteners to work somehow... this is getting ugly and needs cleanup 2010-11-19 Federico Simoncelli knet: add knet_link_timeout and knet_listener_remove 2010-11-18 Fabio M. Di Nitto vty: add concept of peers/hosts sort ring.h header to group functions based on target vty/config: collect handles in relative config structs vty: add basic options for auto-listeners support 2010-11-16 Federico Simoncelli knet: use -1 timeout in epoll to wait indefinitely knet: add acquire/release functions for the knet_listener list 2010-11-15 Fabio M. Di Nitto vty: add write/read config support vty: add config show command also fix assignment again... vty_cli_cmds: Remove unrequired assignments vty_cli_cmds: make help text more generic 2010-11-15 Federico Simoncelli test: add latency/timeout testing comment heartbeat: avoid flaky link when latency is higher than timeout knet: compute latency average 2010-11-12 Federico Simoncelli Merge branch 'master' of github.com:fabbione/kronosnet knet: add link latency statistics (last value) 2010-11-12 Fabio M. Di Nitto vty: add support to configure ip addresses on knet ifaces 2010-11-12 Federico Simoncelli Merge branch 'master' of github.com:fabbione/kronosnet knet: add timestamp into ping packet 2010-11-12 Fabio M. Di Nitto cfg: move ether config in its own struct cfg: drop unrequired locking all config changes are already locked by the vty vty_cli_cmds: fix create interface invokation vty: add mtu config support in interface menu vty_cli_cmds: add a few comments vty_cli_cmds: more sorting cfg: add concept of default mtu and configurable mtu vty_cli_cmds: sort code around move all helper functions at the top and all menu/exec cmd at the bottom 2010-11-11 Fabio M. Di Nitto vty_cli_cmds: set mac address at iface creation time also fix an off-by-one check 2010-11-11 Federico Simoncelli Merge branch 'master' of github.com:fabbione/kronosnet knet: split control thread and heartbeat thread 2010-11-11 Fabio M. Di Nitto vty/cfg: allow multiple parameters per command 2010-11-10 Fabio M. Di Nitto cfg: fix interface name detection this one needs to be precise vty: start adding bindings to core this first snap allows to create/remove knet interfaces 2010-11-10 Federico Simoncelli knet: move debug into ping_test knet: add ping_interval and pong_timeout configuration per link knet: add trivial heartbeat feature cosmetic: fix typo 2010-11-10 Fabio M. Di Nitto whops.. remove debugging/testing code vty_cli: fix tab completion vty_cli: add parameter concept and tab completion 2010-11-09 Fabio M. Di Nitto vty: add basic support for partial command execution and help vty_auth: fix off-by-one in group checking a user with the same name of a group is automatically part of that group 2010-11-04 Fabio M. Di Nitto vty_cli: add ctrl+z handler vty: clear config lock on exit vty_cli_cmds: simply revert action infrastructure knet: check if created tap device has same name of requested vty_cli_cmds: add "no" infrastructure vty_cli_cmds: add interface configuration node layer and fix a segfault in the help function vty_cli_cmds: cosmetic output for who command vty: add configuration handler 2010-11-03 Fabio M. Di Nitto vty_cli: allow to logout via ctrl+d and rename node names to be more useful vty_cli_cmds: add logout/exit commands 2010-11-02 Fabio M. Di Nitto vty_cli_cmds: group code together and make command search non-destructive vty_cli: end of history next should give a clean line 2010-11-01 Fabio M. Di Nitto vty_cli_cmds: add help command vty_cli_cmds: add default help handler vty: handle static allocation consistently vty: laydown basic code to execute commands 2010-10-31 Fabio M. Di Nitto vty_utils: fix segfault when free'ing history 2010-10-29 Fabio M. Di Nitto vty: inform CLI users that the server is going away vty: add note on exit path vty: fix pam memory leakage in case of failure/timeout vty: don't leak memory on pthread_cancel 2010-10-28 Fabio M. Di Nitto cosmetic: add extra return to login info vty_cli: add basic concept of prompt fix line redraw when parsing history vty_cli: fix telnet backchar on x86 vty_cli: add history handler 2010-10-28 Federico Simoncelli vty: cosmetics changes 2010-10-28 Fabio M. Di Nitto vty_cli: add support for insert mode vty_cli: bind extra keys vty_cli: add more home/end escape codes 2010-10-28 Federico Simoncelli vty: add extra escape sequences (home, ins, del, end, pg-up/down) 2010-10-28 Fabio M. Di Nitto vty_cli: add ctrl+t handler vty: fix off by one vty allocation vty_cli: add forward word handler vty_cli: add kill forward word handler vty_cli: add ctrl+w handler vty_cli: fix refresh when adding in the middle of the buffer vty_cli: add backward word handler vty_cli: make add_to_buf echo to terminal temporary disable knet_vty_rewrite_line as it's not used right now this change makes the cli a lot more snappy :) fix an off-by-one buffer init 2010-10-27 Fabio M. Di Nitto vty_cli: add ctrl+u handler vty_cli: add ctrl+h / del handlers vty_cli: add ctrl+e handler vty_cli: add ctrl+k handler vty_cli: add ctrl+c handler temporary change enter handler to differentiate in testing vty_cli: add ctrl+a handler vty_cli: add backward/forward char handlers vty_cli: remove debugging vty_cli: fix parsing of escape chars 2010-10-27 Federico Simoncelli Merge branch 'master' of github.com:fabbione/kronosnet netutils: check name resolution in strtoaddr 2010-10-27 Fabio M. Di Nitto vty_cli: add support for escape chars vty_cli: lay down scheleton for parsing CLI input vty_cli: drop idle update in favour of the generic one vty: make timeout global for all CLI's move the checks into the main vty thread that's idle most of the time anyway. vty cli: add basic infrastructure for CLI vty_utils: some iacs send a response MUST make that dynamic... really 2010-10-27 Federico Simoncelli Merge branch 'master' of github.com:fabbione/kronosnet netutils: add tests for strtoaddr and addrtostr 2010-10-27 Fabio M. Di Nitto vty: configure telnet session and add better error reporting all manual steps at init requires manual checking, the vty loop will do that in an a more elegant fashion :) vty utils: return error if user requested to write more than VTY_MAX_BUFFER_SIZE Probably at somepoint we should allocate this buffer dynamically vty_utils: add connection barrier if socket is closed underneath our feet cosmetic: change welcome message vty: remove debugging setting build: check for pthreads at configure.ac 2010-10-26 Federico Simoncelli knet: add knet_listener structure (replaces knet_handle_bind) 2010-10-25 Fabio M. Di Nitto vty: cleanup error/info reporting code OMG.. did I really write that crap? Merge branch 'master' of github.com:fabbione/kronosnet build: scan for more headers 2010-10-25 Federico Simoncelli knet: do not resolve host names 2010-10-25 Fabio M. Di Nitto vty: free memory 2010-10-22 Fabio M. Di Nitto vty: more fixes on the auth sequence and better error reports Will I ever get this one right? vty_auth: make pam_authenticate errors fatal vty: fix PAM init sequence and error report deal with the lack of /etc/pam.d/kronosnet without crashing vty: log all connections, including source ip and port 2010-10-22 Federico Simoncelli Merge branch 'master' of github.com:fabbione/kronosnet knet: check malloc return code in addrtostr 2010-10-22 Fabio M. Di Nitto vty: init_listener use strtoaddr for the winner! 2010-10-22 Federico Simoncelli Merge branch 'master' of github.com:fabbione/kronosnet knet: add addrtostr 2010-10-22 Fabio M. Di Nitto build: allow default admin group to be configurable and clean up 2010-10-22 Federico Simoncelli Merge branch 'master' of github.com:fabbione/kronosnet knet: improve strtoaddr using getaddrinfo 2010-10-22 Fabio M. Di Nitto Drop log_debug entries for now build: update configure.ac vty: perform group check before allowing users to enable vty: associate username with vty and drastically improve error detection/report also make knet_vty_auth_user reusable 2010-10-21 Fabio M. Di Nitto vty: tell the user on which vty he/she is logging in vty: use struct knet_vty all around vty: address several issues with vty allocation use knet_vty structure as thread arg to pass more data around initialize knet_vtys memory make sure to reuse knet_vtys slots pass data to vty thread clear memory on entry/exit vty: fix set_max_connections sanity checks 2010-10-21 Federico Simoncelli Merge branch 'master' of github.com:fabbione/kronosnet knet: add netutils stub 2010-10-20 Fabio M. Di Nitto vty: add CLI threads and PAM authentication NOTE: /etc/pam.d/kronosnet is now required to telnet to CLI 2010-10-20 Federico Simoncelli knet: add acquire/release functions to lock the host structure list 2010-10-18 Federico Simoncelli knet: use KNET_MAX_EVENTS in epoll_create knet: fix typo knet: use knet_fdset_cloexec, minor lock fix 2010-10-18 Fabio M. Di Nitto Add generic util function to set CLOEXEC on fds 2010-10-18 Federico Simoncelli knet: new knet api interface 2010-10-18 Fabio M. Di Nitto main/vty: return errors from main loop vty: add main loop and bind it to main.c update .gitignore vty: add config options for max connections add also basic scheleton to handle incoming connections build: add default CFLAGS/LDFLAGS for test progs main: parse all command line options add better error handling 2010-10-18 Federico Simoncelli knet: add control thread 2010-10-18 Fabio M. Di Nitto main: add basic daemon infrastructure vty: make vty listener thread safe Add knet_vty_close_listener 2010-10-16 Fabio M. Di Nitto build: fix missing initialization spotted building with F-14 gcc 2010-10-15 Fabio M. Di Nitto vty: fix idiotic use of memory vty: don't use options that are too new... build: add more options to configure.ac vty: add tcp listener 2010-10-15 Federico Simoncelli knet: cleanup, minor fix for printf format knet: add loop count knet: thread-safe list cleanup Merge branch 'master' of github.com:fabbione/kronosnet knet: add thread-safe host list 2010-10-15 Fabio M. Di Nitto ring: fix endian conversion of KNET_FRAME_MAGIC 2010-10-14 Federico Simoncelli knet: add packet inspection placeholder knet: complete rework in network communication 2010-10-14 Fabio M. Di Nitto knet: make knet config functions thread safe also export knet_execute_shell knet: add support to add/remove ip 2010-10-13 Fabio M. Di Nitto knet: cleanup a few things remote unrequired var use knet_close internally avoid unnecessary casts to void * determine ifindex at creation time knet: rework API to allow multiple knet ethernets 2010-10-11 Fabio M. Di Nitto Add dependency on automake and color tests build: add more checks to configure.ac Build test tools automatically 2010-10-11 Federico Simoncelli Add multiple destination hosts for each ring Add ping test, ring test cleanup 2010-10-08 Federico Simoncelli Knet ring test cleanup Merge branch 'master' of github.com:fabbione/kronosnet Add knet_frame structure for ring messages 2010-10-08 Fabio M. Di Nitto knet test: use ip rather than ifconfig that requires /proc mounted knet: add knet_set_up/down 2010-10-08 Federico Simoncelli Use unconnected sockets for ring messages Fix code inconsistencies and variable naming Merge branch 'master' of github.com:fabbione/kronosnet Use INADDR_LOOPBACK macro for localhost 2010-10-08 Fabio M. Di Nitto build: update configure.ac 2010-10-08 Federico Simoncelli Merge branch 'master' of github.com:fabbione/kronosnet Don't use ipv6 by default, use sockaddr_storage in knet_ring structure 2010-10-08 Fabio M. Di Nitto general: clean up usage of log_error knet: add knet_execute_shell and tests this will be used to handle preup/up/down/post-down calls to shell in case the script fails, both stdout and stderr outputs are logged 2010-10-08 Federico Simoncelli Display more information in ring test, minor fixes 2010-10-08 Fabio M. Di Nitto knet: drop redundant checks in set_mtu all those checks are performed by the kernel too, that has a much better idea of what MTU should be. knet: add get/set mac address and update test cases 2010-10-07 Federico Simoncelli Add ring connection and unit test Add syslog option, print error description in log_error 2010-10-07 Fabio M. Di Nitto Add knet_get_mtu/set_mtu functions and add test cases knet: create static knet socket to handle all ioctl work also make ifreq static and global instead of polling everytime. Introduce STATIC macro to access global static variables from tests/ Adjust test case to verify for sock errors Add knet_open and knet_close functions 2010-10-07 Federico Simoncelli Add logging macros 2010-10-07 Fabio M. Di Nitto Clean master to start refactoring 2010-10-07 Federico Simoncelli Identify target node id by mac address 2010-10-06 Federico Simoncelli Set kronosnet hw address using nodeid value Packet tests cleanup Packet inspection should return the node id Fix ipv6 multicast broadcast check Add packet inspection for unicast delivery Improve knet_header and payload allocation Merge branch 'master' of github.com:fabbione/kronosnet Skip documentation build if publican is not available 2010-10-06 Fabio M. Di Nitto debug: add code to decode src/dst mac address so it doesn't get lost... 2010-10-05 Fabio M. Di Nitto fix example config file reduce logging verbosity in main docs: add more format and simpler push to website docs: add basic Makefile.am Add basic publican docs 2010-09-30 Federico Simoncelli Merge branch 'master' of github.com:fabbione/kronosnet Initial handshake implementation 2010-09-30 Fabio M. Di Nitto Revert "Test commit" This reverts commit 7f856ba6006ab644b8bc8fb08c83ac0cd83d0885. Test commit 2010-09-27 Federico Simoncelli Increase socket receive buffer. Extensive optimization in circular buffer algorithm. 2010-09-24 Jan Friesse Fix buffer overflow in sending data Previous dispatch_buffer function added sizeof(struct knet_header) to sended data. This is not needed and causes reading of not initialized data. 2010-09-23 Federico Simoncelli Several changes across the whole tree to use logsys instead of logthread. 2010-09-23 Jan Friesse Process knet_ips and mtu Ability to set link mtu and add ip address(es). Current implementation uses /sbin/ip command. 2010-09-14 Jan Friesse Netdevname again and (hopefully) done right Patch fixes netname option to exists in node config (where it logically belongs). It also introduces config preup function to handle interface pre up (only netdevname is currently copied) Add str_explode function Function supports explode of string to 1-n pieces separated by space or tab. This is useful in convert_ip and also in future parsing of knet_ips. 2010-09-14 Federico Simoncelli Circular buffer code cleanup 2010-09-13 Federico Simoncelli Improve protection against packet duplication (2) 2010-09-10 Federico Simoncelli Improve protection against packet duplication 2010-09-10 Jan Friesse Rename project to Kronosnet 2010-09-10 Federico Simoncelli Merge branch 'master' of github.com:fabbione/clusternet Fix ring buffer cleanup boundary 2010-09-10 Jan Friesse Support for netname change in config file Default tun name can be changed in configuration file. Default name also contains %d string now, so if more than one daemon is running, each device has unique name. 2010-09-09 Federico Simoncelli Use trivial algorithm to clear ring buffer 2010-09-09 Fabio M. Di Nitto Start adding ring buffer to handle packet duplication detection 2010-09-08 Fabio M. Di Nitto Fix check for rollover and add a sequence check the sequence check traps the out-of-order udp packets. We need to fix that with a circular buffer. Add version of cnet_header, src_nodeid and dst_nodeid 2010-09-06 Fabio M. Di Nitto Add 4-rings example 2010-09-03 Fabio M. Di Nitto logging: fix config parsing and make logging conf static Add support for packet sequence rollover 2010-09-01 Fabio M. Di Nitto Go back one step.. disable ping/pong, add redundant-ring support via pckt duplication XXX: must add handling of seq_num rollover or clusternet will halt after uint32_t pkcts Add redundant ring in example.conf Rename variable to something unique add basic (but still useless) heartbeat process Add packet parsing when received on the netsocket Laydown heartbeat thread 2010-08-31 Fabio M. Di Nitto Propagte our nodeid in the pckts Start checking incoming network packages for our MAGIC signature Start adding network control structure also update example.conf to match my basic testing setup Allow command line to override configure file for debug options 2010-08-30 Fabio M. Di Nitto Re-enable IPv6 and remove some more dead code Set standard port Switch to UDP and remove a lot of useless code Bridge incoming connections with the ethernet tested on 4 nodes, almost works :) Detect local IPs and don't iniziate useless connections Associate incoming and outgoing connections XXX: temporary drop IPv6 to preserve life on earth for a few more days remove more paranoia logs Remove some paranoia logging add error and change variable name Start briding FDs around lots of dirty hacks here.. Move RX on eth code in a separate thread don't use MTU to set the buffer, just allocate a static one max ipv6 mtu * 2 change config keyword from "ip" to "nodeips" Update example config we want to configure the interface for the user to avoid lots of complex polling at runtime. 2010-08-27 Fabio M. Di Nitto prepare to deliver pckts to localnet from remote Allocate buffers dynamically based on MTU size Start to create a main loop that's somewhere close to be useful Add netsocket 2010-08-26 Fabio M. Di Nitto add common functions for socket comm and implement status command clusternetd: allow cnet_tool to stop the daemon cnet_tool: laydown structures and allow to send quit command TEST: add simple main loop for testing fix socket handling, add connection to cmd line tool add cmd line tool add basic local commands add local socket data structure control thread: set close on exit and start accepting connections control thread: listen to local socket control thread: add check to see if the thread does init properly start adding the control thread to the daemon it will take care to handle local socket connections from the management tools Add licence/copyright build: update configure.ac for new headers/functions/types add generic utils bit where to add all random useful things nodes: allow selection of inet family for each node 2010-08-25 Fabio M. Di Nitto nodes: fix lookups parameters nodes: allocate conn structs based on data resolution nodes: rename tempip to avoid terms overload parse space separated list of ip addresses starts to add config parsing for nodes and nodes internal data structs 2010-08-16 Fabio M. Di Nitto add example.conf update ignore file conf: start to add node entries config parsing code main: improve drastically error reporting from bottom layers init logging earlier to make config parsers re-usable for reload logging: avoid leaking memory on error main: move messages to DEBUG level config: add support for rerouting and statistics collector config options config: don't double free some bits on error cnet: rename functions config: add missing include cnet: rename files to c(luster)net.{c,h} tun: make it a tap/ether device config: cleanup confdb after usage config: don't leak memory on exit config: reuse corosync.conf format via coroparser and add full config of logging 2010-08-13 Fabio M. Di Nitto add basic config file reading functionality start implementing config file reading clusternetd: add basic stuff