[22:56:59.213] src/tools/ci/cores_backtrace.sh linux /tmp/cores [22:56:59.945] dumping /tmp/cores/postgres-3-23391.core for /tmp/cirrus-ci-build/build/tmp_install/usr/local/pgsql/bin/postgres [22:57:01.816] [New LWP 23391] [22:57:01.816] [New LWP 23255] [22:57:01.816] Downloading separate debug info for /lib/x86_64-linux-gnu/libm.so.6... [22:57:02.589] Downloading separate debug info for /lib/x86_64-linux-gnu/libdl.so.2... [22:57:03.507] Downloading separate debug info for /lib/x86_64-linux-gnu/librt.so.1... [22:57:04.795] Downloading separate debug info for /lib/x86_64-linux-gnu/libgssapi_krb5.so.2... [22:57:07.418] Downloading separate debug info for /lib/x86_64-linux-gnu/libicuuc.so.67... [22:57:07.418] Downloading separate debug info for /.cache/debuginfod_client/f871bbd529a02abd860f0d16b842b5b20234cb49/debuginfo... [22:57:09.501] Downloading separate debug info for /lib/x86_64-linux-gnu/libicui18n.so.67... [22:57:11.024] Downloading separate debug info for /lib/x86_64-linux-gnu/libldap_r-2.4.so.2... [22:57:12.696] Downloading separate debug info for /lib/x86_64-linux-gnu/libxml2.so.2... [22:57:13.820] Downloading separate debug info for /lib/x86_64-linux-gnu/liblz4.so.1... [22:57:14.629] Downloading separate debug info for /lib/x86_64-linux-gnu/libpam.so.0... [22:57:16.003] Downloading separate debug info for /lib/x86_64-linux-gnu/libssl.so.1.1... [22:57:17.743] Downloading separate debug info for /lib/x86_64-linux-gnu/libcrypto.so.1.1... [22:57:19.254] Downloading separate debug info for /lib/x86_64-linux-gnu/libsystemd.so.0... [22:57:20.185] Downloading separate debug info for /lib/x86_64-linux-gnu/libz.so.1... [22:57:21.749] Downloading separate debug info for /lib/x86_64-linux-gnu/libzstd.so.1... [22:57:23.313] Downloading separate debug info for /lib/x86_64-linux-gnu/libpthread.so.0... [22:57:23.784] [Thread debugging using libthread_db enabled] [22:57:23.784] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [22:57:25.047] Downloading separate debug info for /lib64/ld-linux-x86-64.so.2... [22:57:26.484] Downloading separate debug info for /lib/x86_64-linux-gnu/libkrb5.so.3... [22:57:27.552] Downloading separate debug info for /lib/x86_64-linux-gnu/libk5crypto.so.3... [22:57:28.194] Downloading separate debug info for /lib/x86_64-linux-gnu/libcom_err.so.2... [22:57:29.044] Downloading separate debug info for /lib/x86_64-linux-gnu/libkrb5support.so.0... [22:57:31.638] Downloading separate debug info for /lib/x86_64-linux-gnu/libstdc++.so.6... [22:57:32.591] Downloading separate debug info for /lib/x86_64-linux-gnu/libgcc_s.so.1... [22:57:33.432] Downloading separate debug info for /lib/x86_64-linux-gnu/liblber-2.4.so.2... [22:57:34.772] Downloading separate debug info for /lib/x86_64-linux-gnu/libresolv.so.2... [22:57:35.724] Downloading separate debug info for /lib/x86_64-linux-gnu/libsasl2.so.2... [22:57:37.437] Downloading separate debug info for /lib/x86_64-linux-gnu/libgnutls.so.30... [22:57:38.570] Downloading separate debug info for /lib/x86_64-linux-gnu/liblzma.so.5... [22:57:39.388] Downloading separate debug info for /lib/x86_64-linux-gnu/libaudit.so.1... [22:57:40.830] Downloading separate debug info for /lib/x86_64-linux-gnu/libgcrypt.so.20... [22:57:41.479] Downloading separate debug info for /lib/x86_64-linux-gnu/libkeyutils.so.1... [22:57:43.042] Downloading separate debug info for /lib/x86_64-linux-gnu/libp11-kit.so.0... [22:57:43.824] Downloading separate debug info for /lib/x86_64-linux-gnu/libidn2.so.0... [22:57:45.063] Downloading separate debug info for /lib/x86_64-linux-gnu/libunistring.so.2... [22:57:45.882] Downloading separate debug info for /lib/x86_64-linux-gnu/libtasn1.so.6... [22:57:47.011] Downloading separate debug info for /lib/x86_64-linux-gnu/libnettle.so.8... [22:57:47.984] Downloading separate debug info for /lib/x86_64-linux-gnu/libhogweed.so.6... [22:57:49.236] Downloading separate debug info for /lib/x86_64-linux-gnu/libgmp.so.10... [22:57:49.870] Downloading separate debug info for /lib/x86_64-linux-gnu/libcap-ng.so.0... [22:57:50.831] Downloading separate debug info for /lib/x86_64-linux-gnu/libgpg-error.so.0... [22:57:51.627] Downloading separate debug info for /lib/x86_64-linux-gnu/libffi.so.7... [22:57:52.117] Core was generated by `postgres: cascade: startup waiting for 000000020000000000000004 '. [22:57:52.117] Program terminated with signal SIGQUIT, Quit. [22:57:52.117] #0 0x00007f711db41fc7 in sigprocmask () from /lib/x86_64-linux-gnu/libc.so.6 [22:57:52.143] [Current thread is 1 (LWP 23391)] [22:57:52.250] [22:57:52.250] Thread 2 (Thread 0x7f711b4d8a40 (LWP 23255)): [22:57:52.250] #0 0x00007f711dc04a61 in clone () from /lib/x86_64-linux-gnu/libc.so.6 [22:57:52.250] No symbol table info available. [22:57:52.250] #1 0x00007f711dbf2e5e in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [22:57:52.250] No symbol table info available. [22:57:52.250] #2 0x00007f711dbf3497 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [22:57:52.250] No symbol table info available. [22:57:52.250] #3 0x00007f711dbf2cdb in posix_spawn () from /lib/x86_64-linux-gnu/libc.so.6 [22:57:52.250] No symbol table info available. [22:57:52.250] #4 0x00007f711db4ea69 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [22:57:52.250] No symbol table info available. [22:57:52.250] #5 0x0000558387be43e1 in RestoreArchivedFile (path=path@entry=0x7fffc0581ee0 "\200\"X\300\377\177", xlogfname=xlogfname@entry=0x7fffc0582330 "00000002", '0' , "4", recovername=recovername@entry=0x5583886f0a3c "RECOVERYXLOG", expectedSize=16777216, cleanupEnabled=) at ../src/backend/access/transam/xlogarchive.c:177 [22:57:52.250] xlogpath = "pg_wal/RECOVERYXLOG\000\203U\000\000\060OT\210\203U\000\000Awn\210\203U\000\000\071wn\210\203U\000\064\270\035X\300\377\177\000\000\065wn\210\000\000\000\000\030\211M\033q\177\000\000\b\000\000\000\000\000\000\000d\322n\210\203U\000\000\b\036X\300\377\177\000\000X\322n\210\203U\000\000\060OT\210\203U\000\000\060#X\300\377\177\000\000\367\to\210\203U\000\000\b\036X\300\377\177\000\000\365\to\210\203U\000\000\030\211M\033q\177\000\000\214YT\210\203U", '\000' , "\b", '\000' , "\060", '\000' , "\002", '\000' ... [22:57:52.250] xlogRestoreCmd = 0x558389e80958 "cp \"/tmp/cirrus-ci-build/build/testrun/recovery/025_stuck_on_old_timeline/data/t_025_stuck_on_old_timeline_primary_data/archives/00000002", '0' , "4\" \"pg_wal/RECOVERYXLOG\"" [22:57:52.250] lastRestartPointFname = "00000001", '0' , "2\000YT\210\203U", '\000' , "\001\000\000T[\000\000\300\226\315\035q\177\000\000`\227\315\035q\177\000\000T[\000\000\002\000\000\000\000\000\000\000\060\000\000\000\020\355\264\035q\177\000\000\320\026X\300\377\177", '\000' , "W\341\311\035q\177\000\000O\341\311\035q\177\000\000X\t艃U", '\000' , "@", '\000' ... [22:57:52.250] rc = [22:57:52.250] stat_buf = {st_dev = 94023416291604, st_ino = 335544320, st_nlink = 140123806375264, st_mode = 0, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 784066660497414400, st_size = 140736420386864, st_blksize = 94023442762072, st_blocks = 140736420387024, st_atim = {tv_sec = 140736420386608, tv_nsec = 140736420386736}, st_mtim = {tv_sec = 0, tv_nsec = 94023442762072}, st_ctim = {tv_sec = 140123807102103, tv_nsec = 0}, __glibc_reserved = {94023416303408, 140736420389696, 94023418040753}} [22:57:52.250] restartSegNo = [22:57:52.250] restartRedoPtr = 33554472 [22:57:52.250] restartTli = 1 [22:57:52.250] __func__ = "RestoreArchivedFile" [22:57:52.250] #6 0x0000558387bf3071 in XLogFileRead (segno=140736420389600, segno@entry=4, emode=emode@entry=13, tli=tli@entry=2, source=source@entry=XLOG_FROM_ARCHIVE, notfoundOk=notfoundOk@entry=true) at ../src/backend/access/transam/xlogrecovery.c:4220 [22:57:52.250] xlogfname = "00000002", '0' , "4\000\326H\037q\177\000\000\001", '\000' , "\002\000\000\000\bt\261\035q\177\000" [22:57:52.250] activitymsg = "waiting for 00000002", '0' , "4\000\177\000\000\060OT\210\203U\000\000\260`z\210\203U\000\000ȳ\240\210\203U\000\000\004\000\000\t\000\000\000\000\004\000\000\t\000\000\000" [22:57:52.250] path = "\200\"X\300\377\177\000\000\200\"X\300\000\000\000\000\001\000\000\000\000\000\000\000\354MT\210\203U\000\000pg_wal/RECOVERYHISTORY\000\000\b\000\000\000\000\000\000\064\000\000\000\000\000\000\000\000\004TT\210\000\000\000\063\300!X\300\377\177\000\000\b\000\000\000\000\000\000\000d\322n\210\203U\000\000\b\000\000\000\000\000\000\000\071wn\210\203U\000\000h\"X\300\377\177\000\000\065wn\210\203U\000\000\060OT\210\203U\000\000Awn\210\203U\000\000\071wn\210\203U\000\000h\"X\300\377\177\000\000\065wn\210\203U\000\000\030\211M\033q\177\000\000\316XT\210\203U", '\000' ... [22:57:52.250] fd = [22:57:52.250] __func__ = "XLogFileRead" [22:57:52.250] #7 0x0000558387bf3261 in XLogFileReadAnyTLI (segno=4, emode=emode@entry=13, source=XLOG_FROM_ANY) at ../src/backend/access/transam/xlogrecovery.c:4348 [22:57:52.250] hent = [22:57:52.250] tli = 2 [22:57:52.250] cell__state = {l = , i = } [22:57:52.250] path = "\000\000\000\000\000\000\000\000\030\211M\033q\177\000\000p&X\300\377\177\000\000paT\210\203U\000\000\000\000\000\000\000\000\000\000\240\037\000\000\377\377\002\000\000\317\016\211\203U\000\000\064%X\300\377\177\000\000\330_z\210\203U\000\000\221\316\016\211\203U\000\000\000\317\016\211\203U\000\000T%X\300\377\177\000\000\347_z\210\203U\000\000\000\317\016\211\203U\000\000\002\000\000\000\000\000\000\000-iT\210\203U\000\000V%X\300\377\177\000\000T%X\300\377\177", '\000' , "\030\000\000\000\060\000\000\000", '\177' , "\000\000\000\000\000\000\000\000\000\000"... [22:57:52.250] cell = [22:57:52.250] fd = [22:57:52.250] tles = 0x558389e81088 [22:57:52.250] __func__ = "XLogFileReadAnyTLI" [22:57:52.250] #8 0x0000558387bf445d in WaitForWALToBecomeAvailable (RecPtr=RecPtr@entry=67211528, randAccess=false, fetching_ckpt=false, tliRecPtr=tliRecPtr@entry=67211504, replayTLI=2, replayLSN=67211504, nonblocking=false) at ../src/backend/access/transam/xlogrecovery.c:3781 [22:57:52.250] oldSource = [22:57:52.250] startWalReceiver = false [22:57:52.250] last_fail_time = 765759252558161 [22:57:52.250] now = [22:57:52.250] streaming_reply_sent = true [22:57:52.250] __func__ = "WaitForWALToBecomeAvailable" [22:57:52.250] #9 0x0000558387bf4f92 in XLogPageRead (xlogreader=0x558389e2df30, targetPagePtr=67207168, reqLen=4360, targetRecPtr=67211504, readBuf=0x558389e7d128 "\025\321\005") at ../src/backend/access/transam/xlogrecovery.c:3362 [22:57:52.250] private = 0x558389e2df10 [22:57:52.250] emode = 15 [22:57:52.250] targetPageOff = 98304 [22:57:52.250] targetSegNo = 4 [22:57:52.250] r = [22:57:52.250] __func__ = "XLogPageRead" [22:57:52.250] #10 0x0000558387bfd526 in ReadPageInternal (state=state@entry=0x558389e2df30, pageptr=pageptr@entry=67207168, reqLen=reqLen@entry=4360) at ../src/backend/access/transam/xlogreader.c:1060 [22:57:52.250] readLen = [22:57:52.250] targetPageOff = 98304 [22:57:52.250] targetSegNo = 4 [22:57:52.250] hdr = [22:57:52.250] #11 0x0000558387bff181 in XLogDecodeNextRecord (state=state@entry=0x558389e2df30, nonblocking=nonblocking@entry=false) at ../src/backend/access/transam/xlogreader.c:597 [22:57:52.250] RecPtr = 67211504 [22:57:52.250] record = [22:57:52.250] targetPagePtr = 67207168 [22:57:52.250] randAccess = false [22:57:52.250] len = [22:57:52.250] total_len = [22:57:52.250] targetRecOff = 4336 [22:57:52.250] pageHeaderSize = [22:57:52.250] assembled = false [22:57:52.250] gotheader = [22:57:52.250] readOff = [22:57:52.250] decoded = 0x0 [22:57:52.250] errormsg = 0x558387bf1da5 "\351$\375\377\377H\213uhH\205\366t,@\366\306\au&H\215VHH\203\376\270s(H\213FHH\211\342\017\266~8H\211\306\350\223\021\370\377\213T$P\351+\375\377\377H\215=:\211", [22:57:52.250] #12 0x0000558387bffd23 in XLogReadAhead (state=0x558389e2df30, nonblocking=nonblocking@entry=false) at ../src/backend/access/transam/xlogreader.c:973 [22:57:52.250] result = [22:57:52.250] #13 0x0000558387bea52c in XLogPrefetcherNextBlock (pgsr_private=94023442427232, lsn=0x558389e92c58) at ../src/backend/access/transam/xlogprefetcher.c:492 [22:57:52.250] nonblocking = [22:57:52.250] record = [22:57:52.250] prefetcher = 0x558389e2ed60 [22:57:52.250] reader = 0x558389e2df30 [22:57:52.250] replaying_lsn = 67211096 [22:57:52.250] __func__ = "XLogPrefetcherNextBlock" [22:57:52.250] #14 0x0000558387bee8f5 in lrq_prefetch (lrq=) at ../src/backend/access/transam/xlogprefetcher.c:252 [22:57:52.250] No locals. [22:57:52.250] #15 lrq_complete_lsn (lsn=67211504, lrq=0x558389e929d8) at ../src/backend/access/transam/xlogprefetcher.c:290 [22:57:52.250] No locals. [22:57:52.250] #16 XLogPrefetcherReadRecord (prefetcher=prefetcher@entry=0x558389e2ed60, errmsg=errmsg@entry=0x7fffc0586ac8) at ../src/backend/access/transam/xlogprefetcher.c:1037 [22:57:52.250] record = [22:57:52.250] replayed_up_to = 67211504 [22:57:52.250] #17 0x0000558387bf57bf in ReadRecord (xlogprefetcher=0x558389e2ed60, emode=emode@entry=15, fetching_ckpt=fetching_ckpt@entry=false, replayTLI=2) at ../src/backend/access/transam/xlogrecovery.c:3162 [22:57:52.269] errormsg = 0x5583880f565b "AWAVAUATUSH\201\354X\001" [22:57:52.269] record = [22:57:52.269] xlogreader = 0x558389e2df30 [22:57:52.269] private = [22:57:52.269] __func__ = "ReadRecord" [22:57:52.269] #18 0x0000558387bf89e3 in PerformWalRecovery () at ../src/backend/access/transam/xlogrecovery.c:1843 [22:57:52.269] xtime = [22:57:52.269] ru0 = {tv = {tv_sec = 1712444052, tv_usec = 36773}, ru = {ru_utime = {tv_sec = 0, tv_usec = 0}, ru_stime = {tv_sec = 0, tv_usec = 1790}, {ru_maxrss = 16264, __ru_maxrss_word = 16264}, {ru_ixrss = 0, __ru_ixrss_word = 0}, {ru_idrss = 0, __ru_idrss_word = 0}, {ru_isrss = 0, __ru_isrss_word = 0}, {ru_minflt = 404, __ru_minflt_word = 404}, {ru_majflt = 5, __ru_majflt_word = 5}, {ru_nswap = 0, __ru_nswap_word = 0}, {ru_inblock = 0, __ru_inblock_word = 0}, {ru_oublock = 0, __ru_oublock_word = 0}, {ru_msgsnd = 0, __ru_msgsnd_word = 0}, {ru_msgrcv = 0, __ru_msgrcv_word = 0}, {ru_nsignals = 0, __ru_nsignals_word = 0}, {ru_nvcsw = 11, __ru_nvcsw_word = 11}, {ru_nivcsw = 2, __ru_nivcsw_word = 2}}} [22:57:52.269] record = 0x7f711adabe50 [22:57:52.269] reachedRecoveryTarget = false [22:57:52.269] replayTLI = 2 [22:57:52.269] __func__ = "PerformWalRecovery" [22:57:52.269] #19 0x0000558387be2438 in StartupXLOG () at ../src/backend/access/transam/xlog.c:5770 [22:57:52.269] Insert = [22:57:52.269] checkPoint = {redo = 33554472, ThisTimeLineID = 1, PrevTimeLineID = 1, fullPageWrites = true, nextXid = {value = 739}, nextOid = 14202, nextMulti = 1, nextMultiOffset = 0, oldestXid = 731, oldestXidDB = 1, oldestMulti = 1, oldestMultiDB = 1, time = 1712444050, oldestCommitTsXid = 0, newestCommitTsXid = 0, oldestActiveXid = 739} [22:57:52.269] wasShutdown = false [22:57:52.269] didCrash = [22:57:52.269] haveTblspcMap = false [22:57:52.269] haveBackupLabel = true [22:57:52.269] EndOfLog = [22:57:52.269] EndOfLogTLI = [22:57:52.269] newTLI = [22:57:52.269] performedWalRecovery = [22:57:52.269] endOfRecoveryInfo = [22:57:52.269] abortedRecPtr = [22:57:52.269] missingContrecPtr = [22:57:52.269] oldestActiveXID = [22:57:52.269] promoted = false [22:57:52.269] __func__ = "StartupXLOG" [22:57:52.269] #20 0x00005583880a73cc in StartupProcessMain (startup_data=, startup_data_len=) at ../src/backend/postmaster/startup.c:258 [22:57:52.269] No locals. [22:57:52.269] #21 0x000055838809e9e3 in postmaster_child_launch (child_type=child_type@entry=B_STARTUP, startup_data=startup_data@entry=0x0, startup_data_len=startup_data_len@entry=0, client_sock=client_sock@entry=0x0) at ../src/backend/postmaster/launch_backend.c:265 [22:57:52.269] pid = [22:57:52.269] #22 0x00005583880a0a61 in StartChildProcess (type=type@entry=B_STARTUP) at ../src/backend/postmaster/postmaster.c:3928 [22:57:52.269] pid = [22:57:52.269] __func__ = "StartChildProcess" [22:57:52.269] #23 0x00005583880a6b2b in PostmasterMain (argc=argc@entry=4, argv=argv@entry=0x558389e2cdf0) at ../src/backend/postmaster/postmaster.c:1364 [22:57:52.269] opt = [22:57:52.269] status = [22:57:52.269] userDoption = [22:57:52.269] listen_addr_saved = false [22:57:52.269] output_config_variable = [22:57:52.269] __func__ = "PostmasterMain" [22:57:52.269] #24 0x0000558387ec7d15 in main (argc=4, argv=0x558389e2cdf0) at ../src/backend/main/main.c:197 [22:57:52.269] do_check_root = [22:57:52.269] [22:57:52.269] Thread 1 (LWP 23391): [22:57:52.269] #0 0x00007f711db41fc7 in sigprocmask () from /lib/x86_64-linux-gnu/libc.so.6 [22:57:52.269] No symbol table info available. [22:57:52.269] #1 0x00007f711dbf313a in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [22:57:52.269] No symbol table info available. [22:57:52.269] #2 0x00007f711dc04a6f in clone () from /lib/x86_64-linux-gnu/libc.so.6 [22:57:52.269] No symbol table info available.