[11:27:45.067] src/tools/ci/cores_backtrace.sh linux /tmp/cores [11:27:45.803] dumping /tmp/cores/postgres-3-24755.core for /tmp/cirrus-ci-build/build/tmp_install/usr/local/pgsql/bin/postgres [11:27:47.639] [New LWP 24755] [11:27:47.639] [New LWP 24636] [11:27:47.639] Downloading separate debug info for /lib/x86_64-linux-gnu/libm.so.6... [11:27:48.400] Downloading separate debug info for /lib/x86_64-linux-gnu/libdl.so.2... [11:27:49.341] Downloading separate debug info for /lib/x86_64-linux-gnu/librt.so.1... [11:27:50.606] Downloading separate debug info for /lib/x86_64-linux-gnu/libgssapi_krb5.so.2... [11:27:52.976] Downloading separate debug info for /lib/x86_64-linux-gnu/libicuuc.so.67... [11:27:52.976] Downloading separate debug info for /.cache/debuginfod_client/f871bbd529a02abd860f0d16b842b5b20234cb49/debuginfo... [11:27:54.858] Downloading separate debug info for /lib/x86_64-linux-gnu/libicui18n.so.67... [11:27:56.072] Downloading separate debug info for /lib/x86_64-linux-gnu/libldap_r-2.4.so.2... [11:27:57.649] Downloading separate debug info for /lib/x86_64-linux-gnu/libxml2.so.2... [11:27:58.762] Downloading separate debug info for /lib/x86_64-linux-gnu/liblz4.so.1... [11:27:59.559] Downloading separate debug info for /lib/x86_64-linux-gnu/libpam.so.0... [11:28:00.884] Downloading separate debug info for /lib/x86_64-linux-gnu/libssl.so.1.1... [11:28:02.500] Downloading separate debug info for /lib/x86_64-linux-gnu/libcrypto.so.1.1... [11:28:03.980] Downloading separate debug info for /lib/x86_64-linux-gnu/libsystemd.so.0... [11:28:04.893] Downloading separate debug info for /lib/x86_64-linux-gnu/libz.so.1... [11:28:06.427] Downloading separate debug info for /lib/x86_64-linux-gnu/libzstd.so.1... [11:28:07.974] Downloading separate debug info for /lib/x86_64-linux-gnu/libpthread.so.0... [11:28:08.446] [Thread debugging using libthread_db enabled] [11:28:08.446] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [11:28:09.516] Downloading separate debug info for /lib64/ld-linux-x86-64.so.2... [11:28:10.948] Downloading separate debug info for /lib/x86_64-linux-gnu/libkrb5.so.3... [11:28:12.008] Downloading separate debug info for /lib/x86_64-linux-gnu/libk5crypto.so.3... [11:28:12.634] Downloading separate debug info for /lib/x86_64-linux-gnu/libcom_err.so.2... [11:28:13.484] Downloading separate debug info for /lib/x86_64-linux-gnu/libkrb5support.so.0... [11:28:15.768] Downloading separate debug info for /lib/x86_64-linux-gnu/libstdc++.so.6... [11:28:16.689] Downloading separate debug info for /lib/x86_64-linux-gnu/libgcc_s.so.1... [11:28:17.626] Downloading separate debug info for /lib/x86_64-linux-gnu/liblber-2.4.so.2... [11:28:18.907] Downloading separate debug info for /lib/x86_64-linux-gnu/libresolv.so.2... [11:28:19.856] Downloading separate debug info for /lib/x86_64-linux-gnu/libsasl2.so.2... [11:28:21.448] Downloading separate debug info for /lib/x86_64-linux-gnu/libgnutls.so.30... [11:28:22.508] Downloading separate debug info for /lib/x86_64-linux-gnu/liblzma.so.5... [11:28:23.291] Downloading separate debug info for /lib/x86_64-linux-gnu/libaudit.so.1... [11:28:24.675] Downloading separate debug info for /lib/x86_64-linux-gnu/libgcrypt.so.20... [11:28:25.303] Downloading separate debug info for /lib/x86_64-linux-gnu/libkeyutils.so.1... [11:28:26.833] Downloading separate debug info for /lib/x86_64-linux-gnu/libp11-kit.so.0... [11:28:27.596] Downloading separate debug info for /lib/x86_64-linux-gnu/libidn2.so.0... [11:28:28.824] Downloading separate debug info for /lib/x86_64-linux-gnu/libunistring.so.2... [11:28:29.631] Downloading separate debug info for /lib/x86_64-linux-gnu/libtasn1.so.6... [11:28:30.702] Downloading separate debug info for /lib/x86_64-linux-gnu/libnettle.so.8... [11:28:31.629] Downloading separate debug info for /lib/x86_64-linux-gnu/libhogweed.so.6... [11:28:32.850] Downloading separate debug info for /lib/x86_64-linux-gnu/libgmp.so.10... [11:28:33.487] Downloading separate debug info for /lib/x86_64-linux-gnu/libcap-ng.so.0... [11:28:34.403] Downloading separate debug info for /lib/x86_64-linux-gnu/libgpg-error.so.0... [11:28:35.170] Downloading separate debug info for /lib/x86_64-linux-gnu/libffi.so.7... [11:28:35.672] Core was generated by `postgres: cascade: startup waiting for 000000020000000000000004 '. [11:28:35.672] Program terminated with signal SIGQUIT, Quit. [11:28:35.672] #0 0x00007f0b06256fc7 in sigprocmask () from /lib/x86_64-linux-gnu/libc.so.6 [11:28:35.703] [Current thread is 1 (LWP 24755)] [11:28:35.825] [11:28:35.825] Thread 2 (Thread 0x7f0b03beda40 (LWP 24636)): [11:28:35.825] #0 0x00007f0b06319a61 in clone () from /lib/x86_64-linux-gnu/libc.so.6 [11:28:35.825] No symbol table info available. [11:28:35.825] #1 0x00007f0b06307e5e in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [11:28:35.825] No symbol table info available. [11:28:35.825] #2 0x00007f0b06308497 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [11:28:35.825] No symbol table info available. [11:28:35.825] #3 0x00007f0b06307cdb in posix_spawn () from /lib/x86_64-linux-gnu/libc.so.6 [11:28:35.825] No symbol table info available. [11:28:35.825] #4 0x00007f0b06263a69 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [11:28:35.825] No symbol table info available. [11:28:35.825] #5 0x00005582e6986442 in RestoreArchivedFile (path=path@entry=0x7ffc2d89adc0 "`\261\211-\374\177", xlogfname=xlogfname@entry=0x7ffc2d89b210 "00000002", '0' , "4", recovername=recovername@entry=0x5582e74989dc "RECOVERYXLOG", expectedSize=16777216, cleanupEnabled=) at ../src/backend/access/transam/xlogarchive.c:177 [11:28:35.825] xlogpath = "pg_wal/RECOVERYXLOG\000\202U\000\000\201\335.\347\202U\000\000\301\367H\347\202U\000\000\271\367H\347\202U\000\064\230\254\211-\374\177\000\000\265\367H\347\000\000\000\000\030پ\003\v\177\000\000\b\000\000\000\000\000\000\000\314QI\347\202U\000\000謉-\374\177\000\000\300QI\347\202U\000\000\201\335.\347\202U\000\000\020\262\211-\374\177\000\000\227\211I\347\202U\000\000謉-\374\177\000\000\225\211I\347\202U\000\000\030پ\003\v\177\000\000\335\347.\347\202U", '\000' , "\b", '\000' , "\060", '\000' , "\002", '\000' ... [11:28:35.825] xlogRestoreCmd = 0x5582e8a37968 "cp \"/tmp/cirrus-ci-build/build/testrun/recovery/025_stuck_on_old_timeline/data/t_025_stuck_on_old_timeline_primary_data/archives/00000002", '0' , "4\" \"pg_wal/RECOVERYXLOG\"" [11:28:35.825] lastRestartPointFname = "00000001", '0' , "2\000\347.\347\202U", '\000' , "\001\000\000\245`\000\000\300\346>\006\v\177\000\000`\347>\006\v\177\000\000\245`\000\000\002\000\000\000\000\000\000\000\060\000\000\000\020=&\006\v\177\000\000\260\245\211-\374\177", '\000' , "W1;\006\v\177\000\000O1;\006\v\177\000\000hy\243\350\202U", '\000' , "@", '\000' ... [11:28:35.825] rc = [11:28:35.825] stat_buf = {st_dev = 94020712705893, st_ino = 335544320, st_nlink = 139685324483936, st_mode = 0, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 14882962256639247360, st_size = 140721072482064, st_blksize = 94020737137000, st_blocks = 140721072482224, st_atim = {tv_sec = 140721072481808, tv_nsec = 140721072481936}, st_mtim = {tv_sec = 0, tv_nsec = 94020737137000}, st_ctim = {tv_sec = 139685325210775, tv_nsec = 0}, __glibc_reserved = {94020712717697, 140721072484896, 94020714451225}} [11:28:35.825] restartSegNo = [11:28:35.825] restartRedoPtr = 33554472 [11:28:35.825] restartTli = 1 [11:28:35.825] __func__ = "RestoreArchivedFile" [11:28:35.825] #6 0x00005582e69950d2 in XLogFileRead (segno=140721072484800, segno@entry=4, emode=emode@entry=13, tli=tli@entry=2, source=source@entry=XLOG_FROM_ARCHIVE, notfoundOk=notfoundOk@entry=true) at ../src/backend/access/transam/xlogrecovery.c:4209 [11:28:35.825] xlogfname = "00000002", '0' , "4\000\000\000\t", '\000' , "\061\000\000\000\000\000\000\000\220\266\211-\374\177\000\000#&\272\a\v\177\000" [11:28:35.825] activitymsg = "waiting for 00000002", '0' , "4\000\177\000\000\201\335.\347\202U\000\000\360\362T\347\202U\000\000\355\362T\347\202U\000\000(\265\211-\374\177\000\000\353\362T\347\202U\000" [11:28:35.825] path = "`\261\211-\374\177\000\000`\261\211-\000\000\000\000\001\000\000\000\000\000\000\000=\334.\347\202U\000\000pg_wal/RECOVERYHISTORY\000\000\b\000\000\000\000\000\000\064\000\000\000\000\000\000\000\000U\342.\347\000\000\000\063\240\260\211-\374\177\000\000\b\000\000\000\000\000\000\000\314QI\347\202U\000\000\b\000\000\000\000\000\000\000\271\367H\347\202U\000\000H\261\211-\374\177\000\000\265\367H\347\202U\000\000\201\335.\347\202U\000\000\301\367H\347\202U\000\000\271\367H\347\202U\000\000H\261\211-\374\177\000\000\265\367H\347\202U\000\000\030پ\003\v\177\000\000\037\347.\347\202U", '\000' ... [11:28:35.825] fd = [11:28:35.825] __func__ = "XLogFileRead" [11:28:35.825] #7 0x00005582e69952c2 in XLogFileReadAnyTLI (segno=4, emode=emode@entry=13, source=XLOG_FROM_ANY) at ../src/backend/access/transam/xlogrecovery.c:4337 [11:28:35.825] hent = [11:28:35.825] tli = 2 [11:28:35.825] cell__state = {l = , i = } [11:28:35.825] path = "\220\266\211-\374\177\000\000@\266\211-\374\177\000\000\001\000\000\000\000\000\000\000\004\000\000\t", '\000' , "B|\351\347\202U\000\000\300|\351\347\202U\000\000\024\264\211-\374\177\000\000\030\362T\347\202U\000\000\240\037\000\000\377\377\002\000\300|\351\347\202U\000\000\064\264\211-\374\177\000\000'\362T\347\202U\000\000\300|\351\347\202U\000\000\002\000\000\000\000\000\000\000~\367.\347\202U\000\000\066\264\211-\374\177\000\000\064\264\211-\374\177", '\000' , "\030\000\000\000\060\000\000\000\020\264\211-\374\177\000\000W\370.\347\202U\000\000\000\000\000\000\000\000\000\000"... [11:28:35.825] cell = [11:28:35.825] fd = [11:28:35.825] tles = 0x5582e8a38098 [11:28:35.825] __func__ = "XLogFileReadAnyTLI" [11:28:35.825] #8 0x00005582e69964be in WaitForWALToBecomeAvailable (RecPtr=RecPtr@entry=67211528, randAccess=false, fetching_ckpt=false, tliRecPtr=tliRecPtr@entry=67211504, replayTLI=2, replayLSN=67211504, nonblocking=false) at ../src/backend/access/transam/xlogrecovery.c:3770 [11:28:35.825] oldSource = [11:28:35.825] startWalReceiver = false [11:28:35.825] last_fail_time = 767791480759888 [11:28:35.825] now = [11:28:35.825] streaming_reply_sent = true [11:28:35.825] __func__ = "WaitForWALToBecomeAvailable" [11:28:35.825] #9 0x00005582e6996ff3 in XLogPageRead (xlogreader=0x5582e89e4f40, targetPagePtr=67207168, reqLen=4360, targetRecPtr=67211504, readBuf=0x5582e8a34138 "\025\321\005") at ../src/backend/access/transam/xlogrecovery.c:3351 [11:28:35.825] private = 0x5582e89e4f20 [11:28:35.825] emode = 15 [11:28:35.825] targetPageOff = 98304 [11:28:35.825] targetSegNo = 4 [11:28:35.825] r = [11:28:35.825] __func__ = "XLogPageRead" [11:28:35.825] #10 0x00005582e699f44e in ReadPageInternal (state=state@entry=0x5582e89e4f40, pageptr=pageptr@entry=67207168, reqLen=reqLen@entry=4360) at ../src/backend/access/transam/xlogreader.c:1060 [11:28:35.825] readLen = [11:28:35.825] targetPageOff = 98304 [11:28:35.825] targetSegNo = 4 [11:28:35.825] hdr = [11:28:35.825] #11 0x00005582e69a10a9 in XLogDecodeNextRecord (state=state@entry=0x5582e89e4f40, nonblocking=nonblocking@entry=false) at ../src/backend/access/transam/xlogreader.c:597 [11:28:35.825] RecPtr = 67211504 [11:28:35.825] record = [11:28:35.825] targetPagePtr = 67207168 [11:28:35.825] randAccess = false [11:28:35.825] len = [11:28:35.825] total_len = [11:28:35.825] targetRecOff = 4336 [11:28:35.825] pageHeaderSize = [11:28:35.825] assembled = false [11:28:35.825] gotheader = [11:28:35.825] readOff = [11:28:35.825] decoded = 0x0 [11:28:35.825] errormsg = 0x5582e6993e06 "\351$\375\377\377H\213uhH\205\366t,@\366\306\au&H\215VHH\203\376\270s(H\213FHH\211\342\017\266~8H\211\306\350\254\v\370\377\213T$P\351+\375\377\377H\215=\231\355", [11:28:35.825] #12 0x00005582e69a1c4b in XLogReadAhead (state=0x5582e89e4f40, nonblocking=nonblocking@entry=false) at ../src/backend/access/transam/xlogreader.c:973 [11:28:35.825] result = [11:28:35.825] #13 0x00005582e698c58d in XLogPrefetcherNextBlock (pgsr_private=94020736802160, lsn=0x5582e8a49c68) at ../src/backend/access/transam/xlogprefetcher.c:492 [11:28:35.825] nonblocking = [11:28:35.825] record = [11:28:35.825] prefetcher = 0x5582e89e5d70 [11:28:35.825] reader = 0x5582e89e4f40 [11:28:35.825] replaying_lsn = 67211096 [11:28:35.825] __func__ = "XLogPrefetcherNextBlock" [11:28:35.825] #14 0x00005582e6990956 in lrq_prefetch (lrq=) at ../src/backend/access/transam/xlogprefetcher.c:252 [11:28:35.825] No locals. [11:28:35.825] #15 lrq_complete_lsn (lsn=67211504, lrq=0x5582e8a499e8) at ../src/backend/access/transam/xlogprefetcher.c:290 [11:28:35.825] No locals. [11:28:35.825] #16 XLogPrefetcherReadRecord (prefetcher=prefetcher@entry=0x5582e89e5d70, errmsg=errmsg@entry=0x7ffc2d89f9a8) at ../src/backend/access/transam/xlogprefetcher.c:1037 [11:28:35.845] record = [11:28:35.845] replayed_up_to = 67211504 [11:28:35.845] #17 0x00005582e6997820 in ReadRecord (xlogprefetcher=0x5582e89e5d70, emode=emode@entry=15, fetching_ckpt=fetching_ckpt@entry=false, replayTLI=2) at ../src/backend/access/transam/xlogrecovery.c:3151 [11:28:35.845] errormsg = 0x5582e6e9af8c "AWAVAUATUSH\201\354X\001" [11:28:35.845] record = [11:28:35.845] xlogreader = 0x5582e89e4f40 [11:28:35.845] private = [11:28:35.845] __func__ = "ReadRecord" [11:28:35.845] #18 0x00005582e699a9b7 in PerformWalRecovery () at ../src/backend/access/transam/xlogrecovery.c:1832 [11:28:35.845] xtime = [11:28:35.845] ru0 = {tv = {tv_sec = 1714476280, tv_usec = 225364}, ru = {ru_utime = {tv_sec = 0, tv_usec = 2036}, ru_stime = {tv_sec = 0, tv_usec = 0}, {ru_maxrss = 16332, __ru_maxrss_word = 16332}, {ru_ixrss = 0, __ru_ixrss_word = 0}, {ru_idrss = 0, __ru_idrss_word = 0}, {ru_isrss = 0, __ru_isrss_word = 0}, {ru_minflt = 410, __ru_minflt_word = 410}, {ru_majflt = 5, __ru_majflt_word = 5}, {ru_nswap = 0, __ru_nswap_word = 0}, {ru_inblock = 0, __ru_inblock_word = 0}, {ru_oublock = 0, __ru_oublock_word = 0}, {ru_msgsnd = 0, __ru_msgsnd_word = 0}, {ru_msgrcv = 0, __ru_msgrcv_word = 0}, {ru_nsignals = 0, __ru_nsignals_word = 0}, {ru_nvcsw = 11, __ru_nvcsw_word = 11}, {ru_nivcsw = 1, __ru_nivcsw_word = 1}}} [11:28:35.845] record = 0x7f0b034c0e50 [11:28:35.845] reachedRecoveryTarget = false [11:28:35.845] replayTLI = 2 [11:28:35.845] __func__ = "PerformWalRecovery" [11:28:35.845] #19 0x00005582e69843da in StartupXLOG () at ../src/backend/access/transam/xlog.c:5806 [11:28:35.845] Insert = [11:28:35.845] checkPoint = {redo = 33554472, ThisTimeLineID = 1, PrevTimeLineID = 1, fullPageWrites = true, nextXid = {value = 738}, nextOid = 14202, nextMulti = 1, nextMultiOffset = 0, oldestXid = 730, oldestXidDB = 1, oldestMulti = 1, oldestMultiDB = 1, time = 1714476278, oldestCommitTsXid = 0, newestCommitTsXid = 0, oldestActiveXid = 738} [11:28:35.845] wasShutdown = false [11:28:35.845] didCrash = [11:28:35.845] haveTblspcMap = false [11:28:35.845] haveBackupLabel = true [11:28:35.845] EndOfLog = [11:28:35.845] EndOfLogTLI = [11:28:35.845] newTLI = [11:28:35.845] performedWalRecovery = [11:28:35.845] endOfRecoveryInfo = [11:28:35.845] abortedRecPtr = [11:28:35.845] missingContrecPtr = [11:28:35.845] oldestActiveXID = [11:28:35.845] promoted = false [11:28:35.845] __func__ = "StartupXLOG" [11:28:35.845] #20 0x00005582e6e4ccc5 in StartupProcessMain (startup_data=, startup_data_len=) at ../src/backend/postmaster/startup.c:258 [11:28:35.845] No locals. [11:28:35.845] #21 0x00005582e6e442e1 in postmaster_child_launch (child_type=child_type@entry=B_STARTUP, startup_data=startup_data@entry=0x0, startup_data_len=startup_data_len@entry=0, client_sock=client_sock@entry=0x0) at ../src/backend/postmaster/launch_backend.c:265 [11:28:35.845] pid = [11:28:35.845] #22 0x00005582e6e4635a in StartChildProcess (type=type@entry=B_STARTUP) at ../src/backend/postmaster/postmaster.c:3928 [11:28:35.845] pid = [11:28:35.845] __func__ = "StartChildProcess" [11:28:35.845] #23 0x00005582e6e4c424 in PostmasterMain (argc=argc@entry=4, argv=argv@entry=0x5582e89e3e00) at ../src/backend/postmaster/postmaster.c:1364 [11:28:35.845] opt = [11:28:35.845] status = [11:28:35.845] userDoption = [11:28:35.845] listen_addr_saved = false [11:28:35.845] output_config_variable = [11:28:35.845] __func__ = "PostmasterMain" [11:28:35.845] #24 0x00005582e6c6af75 in main (argc=4, argv=0x5582e89e3e00) at ../src/backend/main/main.c:197 [11:28:35.845] do_check_root = [11:28:35.845] [11:28:35.845] Thread 1 (LWP 24755): [11:28:35.845] #0 0x00007f0b06256fc7 in sigprocmask () from /lib/x86_64-linux-gnu/libc.so.6 [11:28:35.845] No symbol table info available. [11:28:35.845] #1 0x00007f0b0630813a in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [11:28:35.845] No symbol table info available. [11:28:35.845] #2 0x00007f0b06319a6f in clone () from /lib/x86_64-linux-gnu/libc.so.6 [11:28:35.845] No symbol table info available.