[09:11:45.333] src/tools/ci/cores_backtrace.sh linux /tmp/cores [09:11:46.075] dumping /tmp/cores/postgres-3-24514.core for /tmp/cirrus-ci-build/build/tmp_install/usr/local/pgsql/bin/postgres [09:11:47.749] [New LWP 24514] [09:11:47.749] [New LWP 24401] [09:11:47.749] Downloading separate debug info for /lib/x86_64-linux-gnu/libm.so.6... [09:11:48.511] Downloading separate debug info for /lib/x86_64-linux-gnu/libdl.so.2... [09:11:49.423] Downloading separate debug info for /lib/x86_64-linux-gnu/librt.so.1... [09:11:50.649] Downloading separate debug info for /lib/x86_64-linux-gnu/libgssapi_krb5.so.2... [09:11:53.020] Downloading separate debug info for /lib/x86_64-linux-gnu/libicuuc.so.67... [09:11:53.020] Downloading separate debug info for /.cache/debuginfod_client/f871bbd529a02abd860f0d16b842b5b20234cb49/debuginfo... [09:11:54.910] Downloading separate debug info for /lib/x86_64-linux-gnu/libicui18n.so.67... [09:11:56.121] Downloading separate debug info for /lib/x86_64-linux-gnu/libldap_r-2.4.so.2... [09:11:57.701] Downloading separate debug info for /lib/x86_64-linux-gnu/libxml2.so.2... [09:11:58.767] Downloading separate debug info for /lib/x86_64-linux-gnu/liblz4.so.1... [09:11:59.573] Downloading separate debug info for /lib/x86_64-linux-gnu/libpam.so.0... [09:12:00.805] Downloading separate debug info for /lib/x86_64-linux-gnu/libssl.so.1.1... [09:12:02.425] Downloading separate debug info for /lib/x86_64-linux-gnu/libcrypto.so.1.1... [09:12:03.904] Downloading separate debug info for /lib/x86_64-linux-gnu/libsystemd.so.0... [09:12:04.817] Downloading separate debug info for /lib/x86_64-linux-gnu/libz.so.1... [09:12:06.350] Downloading separate debug info for /lib/x86_64-linux-gnu/libzstd.so.1... [09:12:07.746] Downloading separate debug info for /lib/x86_64-linux-gnu/libpthread.so.0... [09:12:08.222] [Thread debugging using libthread_db enabled] [09:12:08.222] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [09:12:09.288] Downloading separate debug info for /lib64/ld-linux-x86-64.so.2... [09:12:10.680] Downloading separate debug info for /lib/x86_64-linux-gnu/libkrb5.so.3... [09:12:11.747] Downloading separate debug info for /lib/x86_64-linux-gnu/libk5crypto.so.3... [09:12:12.366] Downloading separate debug info for /lib/x86_64-linux-gnu/libcom_err.so.2... [09:12:13.133] Downloading separate debug info for /lib/x86_64-linux-gnu/libkrb5support.so.0... [09:12:15.421] Downloading separate debug info for /lib/x86_64-linux-gnu/libstdc++.so.6... [09:12:16.348] Downloading separate debug info for /lib/x86_64-linux-gnu/libgcc_s.so.1... [09:12:17.133] Downloading separate debug info for /lib/x86_64-linux-gnu/liblber-2.4.so.2... [09:12:18.048] Downloading separate debug info for /lib/x86_64-linux-gnu/libresolv.so.2... [09:12:18.962] Downloading separate debug info for /lib/x86_64-linux-gnu/libsasl2.so.2... [09:12:20.555] Downloading separate debug info for /lib/x86_64-linux-gnu/libgnutls.so.30... [09:12:21.615] Downloading separate debug info for /lib/x86_64-linux-gnu/liblzma.so.5... [09:12:22.411] Downloading separate debug info for /lib/x86_64-linux-gnu/libaudit.so.1... [09:12:23.791] Downloading separate debug info for /lib/x86_64-linux-gnu/libgcrypt.so.20... [09:12:24.407] Downloading separate debug info for /lib/x86_64-linux-gnu/libkeyutils.so.1... [09:12:25.942] Downloading separate debug info for /lib/x86_64-linux-gnu/libp11-kit.so.0... [09:12:26.706] Downloading separate debug info for /lib/x86_64-linux-gnu/libidn2.so.0... [09:12:27.932] Downloading separate debug info for /lib/x86_64-linux-gnu/libunistring.so.2... [09:12:28.700] Downloading separate debug info for /lib/x86_64-linux-gnu/libtasn1.so.6... [09:12:29.766] Downloading separate debug info for /lib/x86_64-linux-gnu/libnettle.so.8... [09:12:30.689] Downloading separate debug info for /lib/x86_64-linux-gnu/libhogweed.so.6... [09:12:31.911] Downloading separate debug info for /lib/x86_64-linux-gnu/libgmp.so.10... [09:12:32.559] Downloading separate debug info for /lib/x86_64-linux-gnu/libcap-ng.so.0... [09:12:33.479] Downloading separate debug info for /lib/x86_64-linux-gnu/libgpg-error.so.0... [09:12:34.243] Downloading separate debug info for /lib/x86_64-linux-gnu/libffi.so.7... [09:12:34.739] Core was generated by `postgres: cascade: startup waiting for 000000020000000000000004 '. [09:12:34.739] Program terminated with signal SIGQUIT, Quit. [09:12:34.739] #0 0x00007f5350172fc7 in sigprocmask () from /lib/x86_64-linux-gnu/libc.so.6 [09:12:34.770] [Current thread is 1 (LWP 24514)] [09:12:34.890] [09:12:34.890] Thread 2 (Thread 0x7f534db09a40 (LWP 24401)): [09:12:34.890] #0 0x00007f5350235a61 in clone () from /lib/x86_64-linux-gnu/libc.so.6 [09:12:34.890] No symbol table info available. [09:12:34.890] #1 0x00007f5350223e5e in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [09:12:34.890] No symbol table info available. [09:12:34.890] #2 0x00007f5350224497 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [09:12:34.890] No symbol table info available. [09:12:34.890] #3 0x00007f5350223cdb in posix_spawn () from /lib/x86_64-linux-gnu/libc.so.6 [09:12:34.890] No symbol table info available. [09:12:34.890] #4 0x00007f535017fa69 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [09:12:34.890] No symbol table info available. [09:12:34.890] #5 0x0000563e9910eb32 in RestoreArchivedFile (path=path@entry=0x7ffc77e492d0 "p\226\344w\374\177", xlogfname=xlogfname@entry=0x7ffc77e49720 "00000002", '0' , "4", recovername=recovername@entry=0x563e99be45bc "RECOVERYXLOG", expectedSize=16777216, cleanupEnabled=) at ../src/backend/access/transam/xlogarchive.c:177 [09:12:34.890] xlogpath = "pg_wal/RECOVERYXLOG\000>V\000\000}\251\244\231>V\000\000!\263\275\231>V\000\000\031\263\275\231>V\000\064\250\221\344w\374\177\000\000\025\263\275\231\000\000\000\000\030\231\260MS\177\000\000\b\000\000\000\000\000\000\000\234\r\276\231>V\000\000\370\221\344w\374\177\000\000\220\r\276\231>V\000\000}\251\244\231>V\000\000 \227\344w\374\177\000\000wE\276\231>V\000\000\370\221\344w\374\177\000\000uE\276\231>V\000\000\030\231\260MS\177\000\000ٳ\244\231>V", '\000' , "\b", '\000' , "\060", '\000' , "\002", '\000' ... [09:12:34.890] xlogRestoreCmd = 0x563e9a73a958 "cp \"/tmp/cirrus-ci-build/build/testrun/recovery/025_stuck_on_old_timeline/data/t_025_stuck_on_old_timeline_primary_data/archives/00000002", '0' , "4\" \"pg_wal/RECOVERYXLOG\"" [09:12:34.890] lastRestartPointFname = "00000001", '0' , "2\000\263\244\231>V", '\000' , "\001\000\000\261_\000\000\300\246\060PS\177\000\000`\247\060PS\177\000\000\261_\000\000\002\000\000\000\000\000\000\000\060\000\000\000\020\375\027PS\177\000\000\300\212\344w\374\177", '\000' , "W\361,PS\177\000\000O\361,PS\177\000\000X\251s\232>V", '\000' , "@", '\000' ... [09:12:34.890] rc = [09:12:34.890] stat_buf = {st_dev = 94826865654625, st_ino = 335544320, st_nlink = 139995802709344, st_mode = 0, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 18211127860978505216, st_size = 140722319952928, st_blksize = 94826879232344, st_blocks = 140722319953088, st_atim = {tv_sec = 140722319952672, tv_nsec = 140722319952800}, st_mtim = {tv_sec = 0, tv_nsec = 94826879232344}, st_ctim = {tv_sec = 139995803436183, tv_nsec = 0}, __glibc_reserved = {94826865666429, 140722319955760, 94826867330281}} [09:12:34.890] restartSegNo = [09:12:34.890] restartRedoPtr = 33554472 [09:12:34.890] restartTli = 1 [09:12:34.890] __func__ = "RestoreArchivedFile" [09:12:34.890] #6 0x0000563e9911d7c2 in XLogFileRead (segno=140722319955664, segno@entry=4, emode=emode@entry=13, tli=tli@entry=2, source=source@entry=XLOG_FROM_ARCHIVE, notfoundOk=notfoundOk@entry=true) at ../src/backend/access/transam/xlogrecovery.c:4209 [09:12:34.890] xlogfname = "00000002", '0' , "4\000\000\000\000\000\000\000\000\240\233\344w\374\177\000\000#\346\253QS\177\000\000\001", '\000' [09:12:34.890] activitymsg = "waiting for 00000002", '0' , "4\000\177\000\000}\251\244\231>V\000\000\260rə>V\000\000\255rə>V\000\000\070\232\344w\374\177\000\000\310#\356\231>V\000" [09:12:34.890] path = "p\226\344w\374\177\000\000p\226\344w\000\000\000\000\001\000\000\000\000\000\000\000\071\250\244\231>V\000\000pg_wal/RECOVERYHISTORY\000\000\b\000\000\000\000\000\000\064\000\000\000\000\000\000\000\000Q\256\244\231\000\000\000\063\260\225\344w\374\177\000\000\b\000\000\000\000\000\000\000\234\r\276\231>V\000\000\b\000\000\000\000\000\000\000\031\263\275\231>V\000\000X\226\344w\374\177\000\000\025\263\275\231>V\000\000}\251\244\231>V\000\000!\263\275\231>V\000\000\031\263\275\231>V\000\000X\226\344w\374\177\000\000\025\263\275\231>V\000\000\030\231\260MS\177\000\000\033\263\244\231>V", '\000' ... [09:12:34.890] fd = [09:12:34.890] __func__ = "XLogFileRead" [09:12:34.890] #7 0x0000563e9911d9b2 in XLogFileReadAnyTLI (segno=4, emode=emode@entry=13, source=XLOG_FROM_ANY) at ../src/backend/access/transam/xlogrecovery.c:4337 [09:12:34.890] hent = [09:12:34.890] tli = 2 [09:12:34.890] cell__state = {l = , i = } [09:12:34.890] path = "\001\000\000\000\000\000\000\000\004\000\000\t", '\000' , "\275\273\244\231>V\000\000\000\000\000\000\000\000\000\000\242\270Y\232>V\000\000 \271Y\232>V\000\000\240\037\000\000\377\377\002\000\330qə>V\000\000\261\270Y\232>V\000\000 \271Y\232>V\000\000D\231\344w\374\177\000\000\347qə>V\000\000 \271Y\232>V\000\000\002\000\000\000\000\000\000\000zä\231>V\000\000F\231\344w\374\177\000\000D\231\344w\374\177", '\000' , "\030\000\000\000\060\000\000\000 \231\344w\374\177\000\000SĤ\231>V\000\000", '\177' ... [09:12:34.890] cell = [09:12:34.890] fd = [09:12:34.890] tles = 0x563e9a73b088 [09:12:34.890] __func__ = "XLogFileReadAnyTLI" [09:12:34.890] #8 0x0000563e9911ebae in WaitForWALToBecomeAvailable (RecPtr=RecPtr@entry=67211552, randAccess=false, fetching_ckpt=false, tliRecPtr=tliRecPtr@entry=67211528, replayTLI=2, replayLSN=67211528, nonblocking=false) at ../src/backend/access/transam/xlogrecovery.c:3770 [09:12:34.890] oldSource = [09:12:34.890] startWalReceiver = false [09:12:34.890] last_fail_time = 763808934786630 [09:12:34.890] now = [09:12:34.890] streaming_reply_sent = true [09:12:34.890] __func__ = "WaitForWALToBecomeAvailable" [09:12:34.890] #9 0x0000563e9911f6e3 in XLogPageRead (xlogreader=0x563e9a6e7f30, targetPagePtr=67207168, reqLen=4384, targetRecPtr=67211528, readBuf=0x563e9a737128 "\024\321\005") at ../src/backend/access/transam/xlogrecovery.c:3351 [09:12:34.890] private = 0x563e9a6e7f10 [09:12:34.890] emode = 15 [09:12:34.890] targetPageOff = 98304 [09:12:34.890] targetSegNo = 4 [09:12:34.890] r = [09:12:34.890] __func__ = "XLogPageRead" [09:12:34.890] #10 0x0000563e99127b3e in ReadPageInternal (state=state@entry=0x563e9a6e7f30, pageptr=pageptr@entry=67207168, reqLen=reqLen@entry=4384) at ../src/backend/access/transam/xlogreader.c:1060 [09:12:34.890] readLen = [09:12:34.890] targetPageOff = 98304 [09:12:34.890] targetSegNo = 4 [09:12:34.890] hdr = [09:12:34.890] #11 0x0000563e99129799 in XLogDecodeNextRecord (state=state@entry=0x563e9a6e7f30, nonblocking=nonblocking@entry=false) at ../src/backend/access/transam/xlogreader.c:597 [09:12:34.890] RecPtr = 67211528 [09:12:34.890] record = [09:12:34.890] targetPagePtr = 67207168 [09:12:34.890] randAccess = false [09:12:34.890] len = [09:12:34.890] total_len = [09:12:34.890] targetRecOff = 4360 [09:12:34.890] pageHeaderSize = [09:12:34.890] assembled = false [09:12:34.890] gotheader = [09:12:34.890] readOff = [09:12:34.890] decoded = 0x0 [09:12:34.890] errormsg = 0x563e9911c4f6 "\351$\375\377\377H\213uhH\205\366t,@\366\306\au&H\215VHH\203\376\270s(H\213FHH\211\342\017\266~8H\211\306\350\307&\370\377\213T$P\351+\375\377\377H\215=I\244", [09:12:34.890] #12 0x0000563e9912a33b in XLogReadAhead (state=0x563e9a6e7f30, nonblocking=nonblocking@entry=false) at ../src/backend/access/transam/xlogreader.c:973 [09:12:34.890] result = [09:12:34.890] #13 0x0000563e99114c7d in XLogPrefetcherNextBlock (pgsr_private=94826878897504, lsn=0x563e9a74cc58) at ../src/backend/access/transam/xlogprefetcher.c:492 [09:12:34.890] nonblocking = [09:12:34.890] record = [09:12:34.890] prefetcher = 0x563e9a6e8d60 [09:12:34.890] reader = 0x563e9a6e7f30 [09:12:34.890] replaying_lsn = 67211120 [09:12:34.890] __func__ = "XLogPrefetcherNextBlock" [09:12:34.890] #14 0x0000563e99119046 in lrq_prefetch (lrq=) at ../src/backend/access/transam/xlogprefetcher.c:252 [09:12:34.890] No locals. [09:12:34.890] #15 lrq_complete_lsn (lsn=67211528, lrq=0x563e9a74c9d8) at ../src/backend/access/transam/xlogprefetcher.c:290 [09:12:34.890] No locals. [09:12:34.890] #16 XLogPrefetcherReadRecord (prefetcher=prefetcher@entry=0x563e9a6e8d60, errmsg=errmsg@entry=0x7ffc77e4deb8) at ../src/backend/access/transam/xlogprefetcher.c:1037 [09:12:34.890] record = [09:12:34.890] replayed_up_to = 67211528 [09:12:34.890] #17 0x0000563e9911ff10 in ReadRecord (xlogprefetcher=0x563e9a6e8d60, emode=emode@entry=15, fetching_ckpt=fetching_ckpt@entry=false, replayTLI=2) at ../src/backend/access/transam/xlogrecovery.c:3151 [09:12:34.908] errormsg = 0x563e9960bdfd "AWAVAUATUSH\201\354X\001" [09:12:34.908] record = [09:12:34.908] xlogreader = 0x563e9a6e7f30 [09:12:34.908] private = [09:12:34.908] __func__ = "ReadRecord" [09:12:34.908] #18 0x0000563e991230a7 in PerformWalRecovery () at ../src/backend/access/transam/xlogrecovery.c:1832 [09:12:34.908] xtime = [09:12:34.908] ru0 = {tv = {tv_sec = 1710493734, tv_usec = 276803}, ru = {ru_utime = {tv_sec = 0, tv_usec = 2099}, ru_stime = {tv_sec = 0, tv_usec = 0}, {ru_maxrss = 16228, __ru_maxrss_word = 16228}, {ru_ixrss = 0, __ru_ixrss_word = 0}, {ru_idrss = 0, __ru_idrss_word = 0}, {ru_isrss = 0, __ru_isrss_word = 0}, {ru_minflt = 410, __ru_minflt_word = 410}, {ru_majflt = 5, __ru_majflt_word = 5}, {ru_nswap = 0, __ru_nswap_word = 0}, {ru_inblock = 0, __ru_inblock_word = 0}, {ru_oublock = 0, __ru_oublock_word = 0}, {ru_msgsnd = 0, __ru_msgsnd_word = 0}, {ru_msgrcv = 0, __ru_msgrcv_word = 0}, {ru_nsignals = 0, __ru_nsignals_word = 0}, {ru_nvcsw = 11, __ru_nvcsw_word = 11}, {ru_nivcsw = 1, __ru_nivcsw_word = 1}}} [09:12:34.908] record = 0x7f534d3dce68 [09:12:34.908] reachedRecoveryTarget = false [09:12:34.908] replayTLI = 2 [09:12:34.908] __func__ = "PerformWalRecovery" [09:12:34.908] #19 0x0000563e9910cd54 in StartupXLOG () at ../src/backend/access/transam/xlog.c:5723 [09:12:34.908] Insert = [09:12:34.908] checkPoint = {redo = 33554472, ThisTimeLineID = 1, PrevTimeLineID = 1, fullPageWrites = true, nextXid = {value = 738}, nextOid = 14205, nextMulti = 1, nextMultiOffset = 0, oldestXid = 730, oldestXidDB = 1, oldestMulti = 1, oldestMultiDB = 1, time = 1710493732, oldestCommitTsXid = 0, newestCommitTsXid = 0, oldestActiveXid = 738} [09:12:34.908] wasShutdown = false [09:12:34.908] didCrash = [09:12:34.908] haveTblspcMap = false [09:12:34.908] haveBackupLabel = true [09:12:34.908] EndOfLog = [09:12:34.908] EndOfLogTLI = [09:12:34.908] newTLI = [09:12:34.908] performedWalRecovery = [09:12:34.908] endOfRecoveryInfo = [09:12:34.908] abortedRecPtr = [09:12:34.908] missingContrecPtr = [09:12:34.908] oldestActiveXID = [09:12:34.908] promoted = false [09:12:34.908] __func__ = "StartupXLOG" [09:12:34.908] #20 0x0000563e995bdb0c in StartupProcessMain () at ../src/backend/postmaster/startup.c:252 [09:12:34.908] No locals. [09:12:34.908] #21 0x0000563e995ae025 in AuxiliaryProcessMain (auxtype=auxtype@entry=B_STARTUP) at ../src/backend/postmaster/auxprocess.c:91 [09:12:34.908] __func__ = "AuxiliaryProcessMain" [09:12:34.908] #22 0x0000563e995b8dc6 in StartChildProcess (type=type@entry=B_STARTUP) at ../src/backend/postmaster/postmaster.c:5336 [09:12:34.908] pid = [09:12:34.908] __func__ = "StartChildProcess" [09:12:34.908] #23 0x0000563e995bd29a in PostmasterMain (argc=argc@entry=4, argv=argv@entry=0x563e9a6e6df0) at ../src/backend/postmaster/postmaster.c:1470 [09:12:34.908] opt = [09:12:34.908] status = [09:12:34.908] userDoption = [09:12:34.908] listen_addr_saved = false [09:12:34.908] output_config_variable = [09:12:34.908] __func__ = "PostmasterMain" [09:12:34.908] #24 0x0000563e993e806b in main (argc=4, argv=0x563e9a6e6df0) at ../src/backend/main/main.c:197 [09:12:34.908] do_check_root = [09:12:34.908] [09:12:34.908] Thread 1 (LWP 24514): [09:12:34.908] #0 0x00007f5350172fc7 in sigprocmask () from /lib/x86_64-linux-gnu/libc.so.6 [09:12:34.908] No symbol table info available. [09:12:34.908] #1 0x00007f535022413a in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [09:12:34.908] No symbol table info available. [09:12:34.908] #2 0x00007f5350235a6f in clone () from /lib/x86_64-linux-gnu/libc.so.6 [09:12:34.908] No symbol table info available.