]>
mj.ucw.cz Git - libucw.git/log
Martin Mares [Sat, 10 Mar 2007 20:21:23 +0000 (21:21 +0100)]
Unaligned access functions (formerly macros) work in native, big and little endian.
Martin Mares [Sat, 10 Mar 2007 20:00:09 +0000 (21:00 +0100)]
Binary fastbuf functions moved to lib/ff-binary.h.
Martin Mares [Sat, 10 Mar 2007 19:45:38 +0000 (20:45 +0100)]
Removed an unnecessary optimization.
In fact, this could never trigger in recent versions of libucw
as it's already tested in bseek() and bsetpos().
Martin Mares [Sat, 10 Mar 2007 19:44:43 +0000 (20:44 +0100)]
lib/fastbuf.h no longer includes <stdio.h>.
Added zillions of missing includes and fixed handling of EOF.
Mentioned in doc/changes.
Martin Mares [Sat, 10 Mar 2007 19:15:06 +0000 (20:15 +0100)]
Fastbufs now work better on unseekable files.
The seek callback returns success, bseek() and bsetpos() dies if seek
fails, bfilesize() returns -1 if the file is unseekable.
Pavel Charvat [Sat, 10 Mar 2007 12:53:25 +0000 (13:53 +0100)]
use the new mkdist feature in cf/images
Pavel Charvat [Fri, 9 Mar 2007 15:54:07 +0000 (16:54 +0100)]
perl: Added Sherlock::Config::Usage string.
Pavel Charvat [Thu, 8 Mar 2007 12:09:43 +0000 (13:09 +0100)]
Merge with git+ssh://git.ucw.cz/projects/sherlock/GIT/sherlock.git
Pavel Charvat [Thu, 8 Mar 2007 12:05:29 +0000 (13:05 +0100)]
Merge with git+ssh://git.ucw.cz/projects/sherlock/GIT/sherlock.git#v3.10.1
Pavel Charvat [Wed, 7 Mar 2007 15:24:22 +0000 (16:24 +0100)]
ASSERT if now an expression
Pavel Charvat [Wed, 7 Mar 2007 14:24:11 +0000 (15:24 +0100)]
removed EASSERT
Pavel Charvat [Wed, 7 Mar 2007 13:10:12 +0000 (14:10 +0100)]
added always evaluated EASSERT(x)
Pavel Charvat [Wed, 7 Mar 2007 12:08:18 +0000 (13:08 +0100)]
Added 32bit random number generator.
Pavel Charvat [Mon, 5 Mar 2007 20:08:09 +0000 (21:08 +0100)]
small fixes in perl modules
Pavel Charvat [Sat, 3 Mar 2007 13:12:11 +0000 (14:12 +0100)]
partially updated free/mkdist
Pavel Charvat [Thu, 1 Mar 2007 13:31:31 +0000 (14:31 +0100)]
stkstring.h depends on <stdio.h> (sprintf)
Pavel Charvat [Sun, 25 Feb 2007 22:59:33 +0000 (23:59 +0100)]
fixed fcnt_lock interface description
Pavel Charvat [Sun, 25 Feb 2007 22:39:11 +0000 (23:39 +0100)]
added a perl module for fcntl locking
Martin Mares [Sat, 17 Feb 2007 22:06:55 +0000 (23:06 +0100)]
More bug fixes^W^Wimprovements to the timing statistics.
Martin Mares [Sat, 17 Feb 2007 21:18:46 +0000 (22:18 +0100)]
Cleaned up sorter timings and added a final timing statistic.
Martin Mares [Sat, 17 Feb 2007 21:03:32 +0000 (22:03 +0100)]
Updated all users of the timer interface to pass an explicit timer context.
Martin Mares [Sat, 17 Feb 2007 21:02:21 +0000 (22:02 +0100)]
Replaced the old timing functions by something more useful.
get_timer() using global state was making it unusable in many cases,
especially in library routines. I decided to make the callers specify
an explicit context (a timer) instead and re-used the idea of millisecond
timestamps from mainloop.c.
I've moved timestamp_t to config.h, introduced global get_timestamp()
and replaced the timer functions by ones with an explicit timestamp_t *
argument.
Pavel Charvat [Fri, 16 Feb 2007 12:37:11 +0000 (13:37 +0100)]
Merge with git+ssh://git.ucw.cz/projects/sherlock/GIT/sherlock.git
Pavel Charvat [Tue, 13 Feb 2007 11:08:54 +0000 (12:08 +0100)]
Merge with git+ssh://git.ucw.cz/projects/sherlock/GIT/sherlock.git#v3.10.1
Pavel Charvat [Tue, 13 Feb 2007 11:08:13 +0000 (12:08 +0100)]
Merge with git+ssh://git.ucw.cz/projects/sherlock/GIT/sherlock.git#v3.10
Martin Mares [Sat, 10 Feb 2007 21:16:56 +0000 (22:16 +0100)]
Fix calculation of internal sorting buffer and add unification to s-fixint.
Martin Mares [Sat, 10 Feb 2007 21:16:38 +0000 (22:16 +0100)]
Really deallocate the big_buf when radix-splitting.
Martin Mares [Sat, 10 Feb 2007 20:39:58 +0000 (21:39 +0100)]
Created a local TODO list.
Martin Mares [Sat, 10 Feb 2007 20:21:43 +0000 (21:21 +0100)]
Added a magical constant estimating non-uniformity of hash functions.
Martin Mares [Sat, 10 Feb 2007 20:15:36 +0000 (21:15 +0100)]
Fix an off-by-1 error.
Martin Mares [Sat, 10 Feb 2007 19:45:59 +0000 (20:45 +0100)]
Fix bucket estimator.
Martin Mares [Sat, 10 Feb 2007 18:14:31 +0000 (19:14 +0100)]
Radix-sorting is not compatible with custom presort.
Martin Mares [Sat, 10 Feb 2007 18:05:13 +0000 (19:05 +0100)]
Much better decisions on radix sorter parameters.
Martin Mares [Sat, 10 Feb 2007 17:23:58 +0000 (18:23 +0100)]
Added a couple of tests with the old sorter to have reference for speed benchmarks.
(Yes, it horribly duplicates code of sort-test, but I hope it will go away
with the old sorter before merging with mainline.)
Martin Mares [Sat, 10 Feb 2007 13:14:50 +0000 (14:14 +0100)]
Added s-fixint module (internal sorter for small fixed-size records).
Martin Mares [Sat, 10 Feb 2007 11:38:58 +0000 (12:38 +0100)]
Fixed a silly bug.
Martin Mares [Sat, 10 Feb 2007 11:35:33 +0000 (12:35 +0100)]
Moved formatting of sizes to libucw.
Martin Mares [Sat, 10 Feb 2007 11:20:09 +0000 (12:20 +0100)]
Minor improvements of debug messages.
Martin Mares [Fri, 9 Feb 2007 22:58:55 +0000 (23:58 +0100)]
Fixed bug in printing of bucket sizes.
Martin Mares [Fri, 9 Feb 2007 22:41:58 +0000 (23:41 +0100)]
Added a trivial implementation of radix sorting.
Martin Mares [Fri, 9 Feb 2007 21:36:47 +0000 (22:36 +0100)]
Show timings and allow direct I/O.
Martin Mares [Fri, 9 Feb 2007 21:10:36 +0000 (22:10 +0100)]
Remember size of the input file.
Martin Mares [Fri, 9 Feb 2007 21:02:15 +0000 (22:02 +0100)]
All numbers printed by sort-test should be unsigned.
Martin Mares [Fri, 9 Feb 2007 20:37:10 +0000 (21:37 +0100)]
Killed a bug in the tester.
Martin Mares [Fri, 9 Feb 2007 20:26:31 +0000 (21:26 +0100)]
Cannot swap out the source file, because we don't know how to reopen it.
Martin Mares [Fri, 9 Feb 2007 20:21:51 +0000 (21:21 +0100)]
Added a graph-like test case which tests custom presorter and THIS_FB mode.
And as usually fixed some bugs it has uncovered.
Martin Mares [Fri, 9 Feb 2007 18:51:51 +0000 (19:51 +0100)]
Allow empty test cases.
Martin Mares [Fri, 9 Feb 2007 18:48:52 +0000 (19:48 +0100)]
Better messages and sort-test controls.
Martin Mares [Fri, 9 Feb 2007 17:46:47 +0000 (18:46 +0100)]
Added "keep buckets" debug option and fixed some bugs.
Martin Mares [Fri, 9 Feb 2007 17:46:28 +0000 (18:46 +0100)]
A new test.
Martin Mares [Fri, 9 Feb 2007 10:17:00 +0000 (11:17 +0100)]
Merged genconf improvements from mainline.
Martin Mares [Thu, 8 Feb 2007 22:04:56 +0000 (23:04 +0100)]
More elegant evaluation of #if expressions.
Martin Mares [Thu, 8 Feb 2007 22:00:26 +0000 (23:00 +0100)]
Implemented substition of configuration variables.
Martin Mares [Sat, 3 Feb 2007 23:02:10 +0000 (00:02 +0100)]
Added several tests.
These don't include variable-length keys or data yet.
Martin Mares [Sat, 3 Feb 2007 23:01:42 +0000 (00:01 +0100)]
Remember to undefine SORT_PREFIX.
Martin Mares [Sat, 3 Feb 2007 23:01:15 +0000 (00:01 +0100)]
Squash a couple of warnings.
The one about comparison being always the same because of data type
ranges is especially annoying, but I don't know how to shut it up.
Martin Mares [Sat, 3 Feb 2007 12:59:17 +0000 (13:59 +0100)]
Moved low-level operations to a separate file.
Martin Mares [Fri, 2 Feb 2007 22:27:02 +0000 (23:27 +0100)]
Implemented SORT_UNIFY, SORT_UNIQUE and debugged SORT_VAR_DATA.
Now the new sorter is equivalent to the old sorter (i.e., the same algorithms),
but with new interface and standing on new foundations. I will write
a simple test suite and then start filling the gaps with something
interesting.
Martin Mares [Fri, 2 Feb 2007 21:45:07 +0000 (22:45 +0100)]
Empty final bucket should be turned into a file as well.
Martin Mares [Fri, 2 Feb 2007 21:42:00 +0000 (22:42 +0100)]
Cleaned up joining of buckets.
The new bucket semantics really helps.
Martin Mares [Fri, 2 Feb 2007 21:28:43 +0000 (22:28 +0100)]
Implemented swapping in/out buckets.
In some sorting methods, we want to use lots of buckets, but avoid
keeping all the files open and their buffers allocated. The bucket
machinery is now able to swap out a bucket (close its fastbuf) and
then transparently swap it in as needed.
Martin Mares [Fri, 2 Feb 2007 19:41:33 +0000 (20:41 +0100)]
Cleanup of bucket handling.
The bucket operations have much nicer semantics now.
Also added a couple of debug switches.
Martin Mares [Fri, 2 Feb 2007 18:09:59 +0000 (19:09 +0100)]
More bits of the sorter.
In-memory presorting (the general case, but without unification) seems to
work. However, I am not happy with the bucket mechanics yet, it's too
complicated.
Martin Mares [Fri, 2 Feb 2007 13:07:36 +0000 (14:07 +0100)]
Format string cleanup.
Martin Mares [Fri, 2 Feb 2007 12:54:46 +0000 (13:54 +0100)]
big_alloc() now takes a 64-bit argument.
We would like to allocate buffers larger than 4 GB on 64-bit machines.
I've purposefully used u64 instead of size_t, because configuration
variables for buffer sizes will be usually u64's and big_alloc() is
the proper place to report buffer size errors.
Martin Mares [Fri, 2 Feb 2007 12:06:04 +0000 (13:06 +0100)]
Replaced %L by %ll, the former is obsolete and specific to Linux libc5.
Martin Mares [Fri, 2 Feb 2007 11:55:28 +0000 (12:55 +0100)]
Hunted down a dozen of damned printf cast bugs.
Condemned eternally be the man who gave C precise types, but no formatting
sequences for printing them.
Martin Mares [Fri, 2 Feb 2007 11:34:24 +0000 (12:34 +0100)]
Merge with git+ssh://cvs.ucw.cz/projects/sherlock/GIT/sherlock.git
Changes in debug/sorter/retros.c merged manually.
Martin Mares [Fri, 2 Feb 2007 11:31:18 +0000 (12:31 +0100)]
Make !CONFIG_EXACT_CPU work again.
When the CONFIG_EXACT_CPU switch was turned off, the configure script
spat errors on being unable to parse /proc/cpuinfo.
Now we handle the non-exact mode properly by not emitting any model-specific
gcc flags.
Martin Mares [Fri, 2 Feb 2007 11:08:56 +0000 (12:08 +0100)]
The great type cleanup: Use C99 types where applicable.
LibUCW types (u16 and friends) are now based on the C99 <stdint.h> types.
Old-style sbyte, word, sword and addr_int_t are gone (I've removed all refs
to them in the previous commits).
<stddef.h> is now included automatically, bringing types like size_t and
also NULL.
The operations on pointers (GET_P, PUT_P etc.) have been removed from
lib/config.h as nobody uses them and bgetp() is just a duplicate of bgeta()
from lib/fastbuf.h. BYTES_PER_P is gone as well.
#define _GNU_SOURCE had to be moved to lib/config.h, where it belongs anyway.
Martin Mares [Fri, 2 Feb 2007 11:02:39 +0000 (12:02 +0100)]
Use uintptr_t instead of addr_int_t.
No need to use our own type when we already rely on C99 anyway.
Martin Mares [Fri, 2 Feb 2007 10:46:39 +0000 (11:46 +0100)]
Get rid of ancient types: sbyte, word, sword.
s8, u16 and s16 are definitely more readable.
Martin Mares [Wed, 31 Jan 2007 21:45:41 +0000 (22:45 +0100)]
Several bits of the new sorter.
The basic infrastructure seems to fit well. The 2-way merge module is
adapted from the old sorter and it's definitely worth some more optimization.
The sort-test really works :-)
More to come tomorrow.
Martin Mares [Wed, 31 Jan 2007 18:23:50 +0000 (19:23 +0100)]
First bits of the sorter infrastructure.
Configuration is shared with the old sorter.
Martin Mares [Wed, 31 Jan 2007 18:19:37 +0000 (19:19 +0100)]
First attempt at interface of the new sorter.
Martin Mares [Wed, 31 Jan 2007 15:18:52 +0000 (16:18 +0100)]
Merge with git+ssh://cvs.ucw.cz/projects/sherlock/GIT/sherlock.git
Martin Mares [Mon, 29 Jan 2007 11:22:39 +0000 (12:22 +0100)]
Unified printing of flags.
I've moved format_flags() from idxdump to libucw and used it where possible.
I've also unified the direction in which flag bits are printed, so the order
of the state flags has changed [but I don't think anything relies on it].
Pavel Charvat [Wed, 17 Jan 2007 00:38:51 +0000 (01:38 +0100)]
genconf: Stop the compilation if #pipe failed.
Pavel Charvat [Wed, 17 Jan 2007 00:20:56 +0000 (01:20 +0100)]
Support for the following syntax in configuration files:
#if CONFIG_ABC || !CONFIG_DEF
#elsif (CONFIG_GHI && CONFIG_JKL) || CONFIG_MNO
#else
#endif
Alternatively I can cache it to C-like "defined(CONFIG_ABC)"
(CONFIG_ABC alone could expand to its value)
or to pure Perl code without substitutions.
Pavel Charvat [Mon, 15 Jan 2007 10:26:33 +0000 (11:26 +0100)]
Merge with git+ssh://git.ucw.cz/projects/sherlock/GIT/sherlock.git#v3.10.1
Martin Mares [Fri, 12 Jan 2007 11:23:07 +0000 (12:23 +0100)]
Merge with git+ssh://cvs.ucw.cz/projects/sherlock/GIT/sherlock.git
(retros.c merged manually)
Martin Mares [Fri, 12 Jan 2007 11:20:55 +0000 (12:20 +0100)]
CPU_PAGE_SIZE ported from mainline.
Martin Mares [Fri, 12 Jan 2007 11:17:50 +0000 (12:17 +0100)]
Determine system page size in the configure script (introduced CPU_PAGE_SIZE).
This is a work around for broken libc on unstable Debian, but it will
probably be useful elsewhere, too.
Martin Mares [Tue, 2 Jan 2007 14:50:17 +0000 (15:50 +0100)]
Merge with git+ssh://cvs.ucw.cz/projects/sherlock/GIT/sherlock.git
(bigalloc got merged to mainline)
Martin Mares [Tue, 2 Jan 2007 14:23:38 +0000 (15:23 +0100)]
Added missing lib/bigalloc.c from dev-async-shep.
Pavel Charvat [Tue, 2 Jan 2007 14:12:35 +0000 (15:12 +0100)]
Partial merge with dev-async-shep:
- added routines for allocation of large alligned buffers
- improved random-access utility
Pavel Charvat [Wed, 20 Dec 2006 16:17:14 +0000 (17:17 +0100)]
Merge with git+ssh://git.ucw.cz/projects/sherlock/GIT/sherlock.git
Martin Mares [Mon, 18 Dec 2006 13:29:40 +0000 (14:29 +0100)]
Merge with git+ssh://cvs.ucw.cz/projects/sherlock/GIT/sherlock.git
(Incorporated all changes from dev-async-shep, v3.10 and v3.9 through
mainline. All merging went smoothly except for lib/threads.c.)
Pavel Charvat [Mon, 18 Dec 2006 12:23:14 +0000 (13:23 +0100)]
added missing type for a local variable
Martin Mares [Mon, 18 Dec 2006 12:22:00 +0000 (13:22 +0100)]
Fixed a typo preventing compilation in non-threaded mode.
Martin Mares [Mon, 18 Dec 2006 12:15:28 +0000 (13:15 +0100)]
Remove relics of pre-CONFIG_UCW_THREADS times.
Pavel Charvat [Mon, 18 Dec 2006 12:02:50 +0000 (13:02 +0100)]
Merge with git+ssh://git.ucw.cz/projects/sherlock/GIT/sherlock.git
Pavel Charvat [Mon, 18 Dec 2006 12:02:41 +0000 (13:02 +0100)]
Merged mainline with parallel shep-reap:
- bucket file iface supports more than one opened DB
- shep-reap can preload buckets in parallel (useful especially on RAID systems)
Martin Mares [Mon, 18 Dec 2006 11:03:14 +0000 (12:03 +0100)]
Use thread ID for temporary file creation.
Martin Mares [Mon, 18 Dec 2006 10:59:35 +0000 (11:59 +0100)]
Added thread_id to ucwlib_context.
Martin Mares [Mon, 18 Dec 2006 10:52:33 +0000 (11:52 +0100)]
Free thread context structure on thread termination.
Martin Mares [Mon, 18 Dec 2006 11:29:30 +0000 (12:29 +0100)]
Use gettid() on Linux.
This is a hack for two reasons:
o Current glibc releases (up to 2.5) have no wrapper for gettid(),
so we have to use _syscall0 to access it.
o There is no guarantee that pthreads will correspond to kernel
threads. However, in the two existing implementations of pthreads
on Linux, they correspond either to kernel processes or kernel
threads, so assumptions that the (pid, tid) pair is unique are
correct.
On the other hand, using real thread ID's has many advantages,
so I am keeping this patch at least in the dev-sorter branch to see
if it's safe or not.
Martin Mares [Mon, 18 Dec 2006 11:03:14 +0000 (12:03 +0100)]
Use thread ID for temporary file creation.
Martin Mares [Mon, 18 Dec 2006 10:59:35 +0000 (11:59 +0100)]
Added thread_id to ucwlib_context.
Martin Mares [Mon, 18 Dec 2006 10:52:33 +0000 (11:52 +0100)]
Free thread context structure on thread termination.